2025-07-17T08:42:28.7146901Z Current runner version: '2.326.0'
2025-07-17T08:42:28.7153042Z Runner name: 'gpu4f67'
2025-07-17T08:42:28.7153753Z Runner group name: 'linux.rocm.gpu.group'
2025-07-17T08:42:28.7154659Z Machine name: 'gpu4f67'
2025-07-17T08:42:28.7157591Z ##[group]GITHUB_TOKEN Permissions
2025-07-17T08:42:28.7159793Z Contents: read
2025-07-17T08:42:28.7160351Z Metadata: read
2025-07-17T08:42:28.7161000Z ##[endgroup]
2025-07-17T08:42:28.7163365Z Secret source: Actions
2025-07-17T08:42:28.7164325Z Prepare workflow directory
2025-07-17T08:42:29.1623292Z Prepare all required actions
2025-07-17T08:42:29.1668119Z Getting action download info
2025-07-17T08:42:29.5345617Z Download action repository 'pytorch/pytorch@main' (SHA:39ac189808c61588f3594dbc2fc1d69bb6194c47)
2025-07-17T08:42:36.0849455Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722)
2025-07-17T08:42:36.7183443Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076)
2025-07-17T08:42:37.1868420Z Download action repository 'pytorch/test-infra@main' (SHA:a9ec424ad5e5851e47d68139cfd953b4031778d5)
2025-07-17T08:42:38.1393907Z Download action repository 'actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-07-17T08:42:38.8417137Z Getting action download info
2025-07-17T08:42:38.9691116Z Download action repository 'actions/checkout@v4' (SHA:11bd71901bbe5b1630ceea73d27597364c9af683)
2025-07-17T08:42:39.6245407Z Getting action download info
2025-07-17T08:42:39.7516407Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2025-07-17T08:42:40.2309226Z Getting action download info
2025-07-17T08:42:40.4271563Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/main (a38f433be2e94a64b095a44ba39879d02d0c2316)
2025-07-17T08:42:40.4276259Z ##[group] Inputs
2025-07-17T08:42:40.4276625Z   build-environment: linux-jammy-rocm-py3.10
2025-07-17T08:42:40.4277323Z   test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}]}
2025-07-17T08:42:40.4278380Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:40.4279048Z   sync-tag: 
2025-07-17T08:42:40.4279941Z   timeout-minutes: 300
2025-07-17T08:42:40.4280203Z   tests-to-include: 
2025-07-17T08:42:40.4280430Z   dashboard-tag: 
2025-07-17T08:42:40.4280981Z   disable-monitor: true
2025-07-17T08:42:40.4281267Z   monitor-log-interval: 5
2025-07-17T08:42:40.4281569Z   monitor-data-collect-interval: 1
2025-07-17T08:42:40.4281841Z ##[endgroup]
2025-07-17T08:42:40.4282200Z Complete job name: rocm-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2)
2025-07-17T08:42:40.6312243Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2025-07-17T08:42:40.6312993Z with:
2025-07-17T08:42:40.6313187Z   no-sudo: true
2025-07-17T08:42:40.6313382Z   submodules: recursive
2025-07-17T08:42:40.6313589Z   fetch-depth: 0
2025-07-17T08:42:40.6313927Z env:
2025-07-17T08:42:40.6314118Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:40.6314350Z ##[endgroup]
2025-07-17T08:42:40.6399853Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-07-17T08:42:40.6400722Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-07-17T08:42:40.6432876Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:40.6433226Z env:
2025-07-17T08:42:40.6433406Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:40.6433645Z ##[endgroup]
2025-07-17T08:42:40.6666072Z ##[group]Run # Use all available CPUs for fetching
2025-07-17T08:42:40.6666795Z # Use all available CPUs for fetching
2025-07-17T08:42:40.6667088Z cd "${GITHUB_WORKSPACE}"
2025-07-17T08:42:40.6667386Z git config --global fetch.parallel 0
2025-07-17T08:42:40.6667713Z git config --global submodule.fetchJobs 0
2025-07-17T08:42:40.6668002Z 
2025-07-17T08:42:40.6668298Z # Clean workspace. The default checkout action should also do this, but
2025-07-17T08:42:40.6668687Z # do it here as well just in case
2025-07-17T08:42:40.6668958Z if [[ -d .git ]]; then
2025-07-17T08:42:40.6669194Z   if [ -z "${NO_SUDO}" ]; then
2025-07-17T08:42:40.6669461Z     sudo git clean -ffdx
2025-07-17T08:42:40.6669688Z   else
2025-07-17T08:42:40.6669890Z     git clean -ffdx
2025-07-17T08:42:40.6670105Z   fi
2025-07-17T08:42:40.6670295Z fi
2025-07-17T08:42:40.6695339Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:40.6695677Z env:
2025-07-17T08:42:40.6695882Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:40.6696201Z   NO_SUDO: true
2025-07-17T08:42:40.6696485Z ##[endgroup]
2025-07-17T08:42:41.2229648Z Removing .additional_ci_files/
2025-07-17T08:42:41.2230094Z Removing build/
2025-07-17T08:42:41.2230306Z Removing dist/
2025-07-17T08:42:41.2230540Z Removing test/test-reports/
2025-07-17T08:42:41.2295704Z ##[group]Run actions/checkout@v4
2025-07-17T08:42:41.2296060Z with:
2025-07-17T08:42:41.2296291Z   ref: a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:42:41.2296580Z   fetch-depth: 0
2025-07-17T08:42:41.2296788Z   submodules: recursive
2025-07-17T08:42:41.2297019Z   show-progress: false
2025-07-17T08:42:41.2297246Z   repository: pytorch/pytorch
2025-07-17T08:42:41.2297612Z   token: ***
2025-07-17T08:42:41.2297799Z   ssh-strict: true
2025-07-17T08:42:41.2298002Z   ssh-user: git
2025-07-17T08:42:41.2298212Z   persist-credentials: true
2025-07-17T08:42:41.2298435Z   clean: true
2025-07-17T08:42:41.2298673Z   sparse-checkout-cone-mode: true
2025-07-17T08:42:41.2298939Z   fetch-tags: false
2025-07-17T08:42:41.2299250Z   lfs: false
2025-07-17T08:42:41.2299509Z   set-safe-directory: true
2025-07-17T08:42:41.2299858Z env:
2025-07-17T08:42:41.2300263Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:41.2300595Z ##[endgroup]
2025-07-17T08:42:41.3483806Z Syncing repository: pytorch/pytorch
2025-07-17T08:42:41.3485074Z ##[group]Getting Git version info
2025-07-17T08:42:41.3485552Z Working directory is '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch'
2025-07-17T08:42:41.3486168Z [command]/usr/bin/git version
2025-07-17T08:42:41.3500429Z git version 2.34.1
2025-07-17T08:42:41.3525975Z ##[endgroup]
2025-07-17T08:42:41.3543788Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/b5a18243-96e7-4d95-a69e-fc211c675940/.gitconfig'
2025-07-17T08:42:41.3553637Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/b5a18243-96e7-4d95-a69e-fc211c675940' before making global git config changes
2025-07-17T08:42:41.3554522Z Adding repository directory to the temporary git global config as a safe directory
2025-07-17T08:42:41.3557773Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-07-17T08:42:41.3603435Z [command]/usr/bin/git config --local --get remote.origin.url
2025-07-17T08:42:41.3636615Z https://github.com/pytorch/pytorch
2025-07-17T08:42:41.3651594Z ##[group]Removing previously created refs, to avoid conflicts
2025-07-17T08:42:41.3654613Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD
2025-07-17T08:42:41.3682167Z HEAD
2025-07-17T08:42:41.3730114Z ##[endgroup]
2025-07-17T08:42:41.3732367Z [command]/usr/bin/git submodule status
2025-07-17T08:42:41.4188561Z  7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe)
2025-07-17T08:42:41.4317912Z  4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081)
2025-07-17T08:42:41.4437824Z  b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327)
2025-07-17T08:42:41.4575597Z  c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0)
2025-07-17T08:42:41.4649307Z  2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07 third_party/NVTX (v3.1.0-263-g2942f16)
2025-07-17T08:42:41.4757858Z  1d8f600fd424278486eade7ed3e877c99f0846b1 third_party/VulkanMemoryAllocator (v2.1.0-982-g1d8f600)
2025-07-17T08:42:41.5333816Z  51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a0103656)
2025-07-17T08:42:41.5388479Z  01aae101b9e5e94d6c16a9514c9fb8df99c93150 third_party/aiter (v0.1.1-92-g01aae101)
2025-07-17T08:42:41.5429310Z  299e5928955cc62af9968370293b916f5130916f third_party/benchmark (v1.9.3)
2025-07-17T08:42:41.5491875Z  434d19f696da62c12b5372b32cbc9ba968588d7e third_party/composable_kernel (rocm-6.4.1-172-g434d19f69)
2025-07-17T08:42:41.5656469Z  3af7f2c16147f3fbc6e4d717032daf505dc1652c third_party/cpp-httplib (v0.20.1)
2025-07-17T08:42:41.5827937Z  5e3d2445e6a84d9599bee2bf78edbb4d80865e1d third_party/cpuinfo (5e3d244)
2025-07-17T08:42:41.5880872Z  f937055efc6d414d11f4c6577e3977fe74f35fb6 third_party/cudnn_frontend (v0.5-52-gf937055)
2025-07-17T08:42:41.6010484Z  b995f933179c22d3fe0d871c3a53d11e4681950f third_party/cutlass (v4.0.0)
2025-07-17T08:42:41.6074763Z  157e88b750c452bef2ab4653fe9d1eeb151ce4c3 third_party/fbgemm (v1.2.0-186-g157e88b7)
2025-07-17T08:42:41.6187817Z  979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4)
2025-07-17T08:42:41.6223421Z  a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757 third_party/flatbuffers (v24.12.23)
2025-07-17T08:42:41.6661031Z  40626af88bd7df9a5fb80be7b25ac85b122d6c21 third_party/fmt (11.2.0)
2025-07-17T08:42:41.6816101Z  3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17)
2025-07-17T08:42:41.6959143Z  c7b7b022c124d9643957d9bd55f57ac59fce8fa2 third_party/gloo (remotes/origin/HEAD)
2025-07-17T08:42:41.7210938Z  52eb8108c5bdec04579160ae17225d66034bd723 third_party/googletest (release-1.8.0-3544-g52eb8108)
2025-07-17T08:42:41.7321046Z  719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1)
2025-07-17T08:42:41.7410616Z  dec1d23ca65ab069d225dfe40dea14f455170959 third_party/ittapi (v3.25.5)
2025-07-17T08:42:41.7672813Z  5e7501833f1021ce6f618572d3baf657b6319658 third_party/kineto (remotes/origin/HEAD)
2025-07-17T08:42:41.7713042Z  cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7 third_party/kleidiai (v1.8.0)
2025-07-17T08:42:41.7751890Z  fbd8b99c2b828428947d70fdc046bb55609be93e third_party/mimalloc (v2.2.4)
2025-07-17T08:42:41.7789514Z  55f93686c01528224f448c19128836e7df245f72 third_party/nlohmann (v3.12.0)
2025-07-17T08:42:41.8148171Z  e709452ef2bbc1d113faf678c24e6d3467696e83 third_party/onnx (v1.18.0)
2025-07-17T08:42:41.8188037Z  a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2)
2025-07-17T08:42:41.8231214Z  0fa0ef591e38c2758e3184c6c23e497b9f732ffa third_party/pocketfft (release_for_eigen-40-g0fa0ef5)
2025-07-17T08:42:41.8650024Z  d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b)
2025-07-17T08:42:41.8777401Z  072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master)
2025-07-17T08:42:41.8858397Z  4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e)
2025-07-17T08:42:41.8923510Z  a2e59f0e7065404b44dfe92a28aca47ba1378dc4 third_party/pybind11 (v2.11.0-182-ga2e59f0e)
2025-07-17T08:42:41.9049158Z  f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated)
2025-07-17T08:42:41.9158268Z  5a1d179df9cf652951b59010a2d2075372d67f68 third_party/sleef (3.8)
2025-07-17T08:42:41.9282166Z  52791a2fd214b2a9dc5759d36725909c1daa7f2e third_party/tensorpipe (remotes/origin/master)
2025-07-17T08:42:41.9302955Z ##[group]Cleaning the repository
2025-07-17T08:42:41.9305867Z [command]/usr/bin/git clean -ffdx
2025-07-17T08:42:41.9685621Z [command]/usr/bin/git reset --hard HEAD
2025-07-17T08:42:42.1736188Z HEAD is now at 7770f763058 Update on "[hop] set capture_scalar_outputs=True by default for compiled hops"
2025-07-17T08:42:42.1818531Z ##[endgroup]
2025-07-17T08:42:42.1819765Z ##[group]Disabling automatic garbage collection
2025-07-17T08:42:42.1824099Z [command]/usr/bin/git config --local gc.auto 0
2025-07-17T08:42:42.1865772Z ##[endgroup]
2025-07-17T08:42:42.1866466Z ##[group]Setting up auth
2025-07-17T08:42:42.1871391Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-07-17T08:42:42.1913138Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-07-17T08:42:42.2306378Z Entering 'android/libs/fbjni'
2025-07-17T08:42:42.2382166Z Entering 'third_party/FP16'
2025-07-17T08:42:42.2455056Z Entering 'third_party/FXdiv'
2025-07-17T08:42:42.2529434Z Entering 'third_party/NNPACK'
2025-07-17T08:42:42.2599026Z Entering 'third_party/NVTX'
2025-07-17T08:42:42.2675593Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-17T08:42:42.2752867Z Entering 'third_party/XNNPACK'
2025-07-17T08:42:42.2835322Z Entering 'third_party/aiter'
2025-07-17T08:42:42.2913912Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-17T08:42:42.2999311Z Entering 'third_party/benchmark'
2025-07-17T08:42:42.3073493Z Entering 'third_party/composable_kernel'
2025-07-17T08:42:42.3155221Z Entering 'third_party/cpp-httplib'
2025-07-17T08:42:42.3233683Z Entering 'third_party/cpuinfo'
2025-07-17T08:42:42.3307401Z Entering 'third_party/cudnn_frontend'
2025-07-17T08:42:42.3378453Z Entering 'third_party/cutlass'
2025-07-17T08:42:42.3461363Z Entering 'third_party/fbgemm'
2025-07-17T08:42:42.3534411Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-17T08:42:42.3600878Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-17T08:42:42.3675050Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-17T08:42:42.3746481Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-17T08:42:42.3816085Z Entering 'third_party/fbgemm/external/googletest'
2025-07-17T08:42:42.3882242Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-17T08:42:42.3947002Z Entering 'third_party/fbgemm/external/json'
2025-07-17T08:42:42.4025477Z Entering 'third_party/flash-attention'
2025-07-17T08:42:42.4092846Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-17T08:42:42.4168331Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-17T08:42:42.4254478Z Entering 'third_party/flatbuffers'
2025-07-17T08:42:42.4327598Z Entering 'third_party/fmt'
2025-07-17T08:42:42.4401367Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-17T08:42:42.4468550Z Entering 'third_party/gloo'
2025-07-17T08:42:42.4548640Z Entering 'third_party/googletest'
2025-07-17T08:42:42.4628096Z Entering 'third_party/ideep'
2025-07-17T08:42:42.4705563Z Entering 'third_party/ideep/mkl-dnn'
2025-07-17T08:42:42.4782924Z Entering 'third_party/ittapi'
2025-07-17T08:42:42.4855500Z Entering 'third_party/kineto'
2025-07-17T08:42:42.4924655Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-17T08:42:42.4989448Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-17T08:42:42.5063858Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-17T08:42:42.5123182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-17T08:42:42.5189678Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-17T08:42:42.5251680Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-17T08:42:42.5326108Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-17T08:42:42.5397689Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-17T08:42:42.5468685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-17T08:42:42.5538267Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-17T08:42:42.5609111Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-17T08:42:42.5672027Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-17T08:42:42.5744190Z Entering 'third_party/kleidiai'
2025-07-17T08:42:42.5810114Z Entering 'third_party/mimalloc'
2025-07-17T08:42:42.5884746Z Entering 'third_party/nlohmann'
2025-07-17T08:42:42.5956190Z Entering 'third_party/onnx'
2025-07-17T08:42:42.6048666Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-17T08:42:42.6128081Z Entering 'third_party/opentelemetry-cpp'
2025-07-17T08:42:42.6200423Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-17T08:42:42.6268957Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-17T08:42:42.6332478Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-17T08:42:42.6399136Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-17T08:42:42.6471579Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-17T08:42:42.6532837Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-17T08:42:42.6609647Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-17T08:42:42.6672433Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-17T08:42:42.6743589Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-17T08:42:42.6814273Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-17T08:42:42.6910096Z Entering 'third_party/pocketfft'
2025-07-17T08:42:42.6980409Z Entering 'third_party/protobuf'
2025-07-17T08:42:42.7052024Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-17T08:42:42.7121064Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-17T08:42:42.7196100Z Entering 'third_party/psimd'
2025-07-17T08:42:42.7273105Z Entering 'third_party/pthreadpool'
2025-07-17T08:42:42.7352135Z Entering 'third_party/pybind11'
2025-07-17T08:42:42.7415827Z Entering 'third_party/python-peachpy'
2025-07-17T08:42:42.7488558Z Entering 'third_party/sleef'
2025-07-17T08:42:42.7557732Z Entering 'third_party/tensorpipe'
2025-07-17T08:42:42.7632645Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-17T08:42:42.7695160Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-17T08:42:42.7764674Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-17T08:42:42.7834875Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-17T08:42:42.7900748Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-17T08:42:42.8005846Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-07-17T08:42:42.8042712Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-07-17T08:42:42.8414622Z Entering 'android/libs/fbjni'
2025-07-17T08:42:42.8483717Z Entering 'third_party/FP16'
2025-07-17T08:42:42.8555297Z Entering 'third_party/FXdiv'
2025-07-17T08:42:42.8628658Z Entering 'third_party/NNPACK'
2025-07-17T08:42:42.8706130Z Entering 'third_party/NVTX'
2025-07-17T08:42:42.8786668Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-17T08:42:42.8861035Z Entering 'third_party/XNNPACK'
2025-07-17T08:42:42.8952402Z Entering 'third_party/aiter'
2025-07-17T08:42:42.9026982Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-17T08:42:42.9112417Z Entering 'third_party/benchmark'
2025-07-17T08:42:42.9187285Z Entering 'third_party/composable_kernel'
2025-07-17T08:42:42.9276292Z Entering 'third_party/cpp-httplib'
2025-07-17T08:42:42.9350761Z Entering 'third_party/cpuinfo'
2025-07-17T08:42:42.9427956Z Entering 'third_party/cudnn_frontend'
2025-07-17T08:42:42.9501960Z Entering 'third_party/cutlass'
2025-07-17T08:42:42.9581809Z Entering 'third_party/fbgemm'
2025-07-17T08:42:42.9656021Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-17T08:42:42.9722088Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-17T08:42:42.9798074Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-17T08:42:42.9869201Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-17T08:42:42.9951927Z Entering 'third_party/fbgemm/external/googletest'
2025-07-17T08:42:43.0017616Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-17T08:42:43.0082815Z Entering 'third_party/fbgemm/external/json'
2025-07-17T08:42:43.0158648Z Entering 'third_party/flash-attention'
2025-07-17T08:42:43.0231318Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-17T08:42:43.0316327Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-17T08:42:43.0404582Z Entering 'third_party/flatbuffers'
2025-07-17T08:42:43.0481494Z Entering 'third_party/fmt'
2025-07-17T08:42:43.0553188Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-17T08:42:43.0622272Z Entering 'third_party/gloo'
2025-07-17T08:42:43.0691326Z Entering 'third_party/googletest'
2025-07-17T08:42:43.0758923Z Entering 'third_party/ideep'
2025-07-17T08:42:43.0828499Z Entering 'third_party/ideep/mkl-dnn'
2025-07-17T08:42:43.0904996Z Entering 'third_party/ittapi'
2025-07-17T08:42:43.0968743Z Entering 'third_party/kineto'
2025-07-17T08:42:43.1038456Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-17T08:42:43.1102734Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-17T08:42:43.1169035Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-17T08:42:43.1233702Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-17T08:42:43.1301812Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-17T08:42:43.1364269Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-17T08:42:43.1437313Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-17T08:42:43.1510118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-17T08:42:43.1572184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-17T08:42:43.1642057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-17T08:42:43.1716304Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-17T08:42:43.1783889Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-17T08:42:43.1853282Z Entering 'third_party/kleidiai'
2025-07-17T08:42:43.1924766Z Entering 'third_party/mimalloc'
2025-07-17T08:42:43.2000489Z Entering 'third_party/nlohmann'
2025-07-17T08:42:43.2073296Z Entering 'third_party/onnx'
2025-07-17T08:42:43.2170436Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-17T08:42:43.2250366Z Entering 'third_party/opentelemetry-cpp'
2025-07-17T08:42:43.2322522Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-17T08:42:43.2392017Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-17T08:42:43.2460255Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-17T08:42:43.2526992Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-17T08:42:43.2596507Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-17T08:42:43.2665429Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-17T08:42:43.2728958Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-17T08:42:43.2793660Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-17T08:42:43.2856599Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-17T08:42:43.2930868Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-17T08:42:43.3022613Z Entering 'third_party/pocketfft'
2025-07-17T08:42:43.3094889Z Entering 'third_party/protobuf'
2025-07-17T08:42:43.3171863Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-17T08:42:43.3242372Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-17T08:42:43.3318663Z Entering 'third_party/psimd'
2025-07-17T08:42:43.3393871Z Entering 'third_party/pthreadpool'
2025-07-17T08:42:43.3464248Z Entering 'third_party/pybind11'
2025-07-17T08:42:43.3529691Z Entering 'third_party/python-peachpy'
2025-07-17T08:42:43.3598349Z Entering 'third_party/sleef'
2025-07-17T08:42:43.3669694Z Entering 'third_party/tensorpipe'
2025-07-17T08:42:43.3736207Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-17T08:42:43.3797919Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-17T08:42:43.3865077Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-17T08:42:43.3927504Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-17T08:42:43.3995587Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-17T08:42:43.4094929Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-07-17T08:42:43.4140011Z ##[endgroup]
2025-07-17T08:42:43.4140481Z ##[group]Fetching the repository
2025-07-17T08:42:43.4147167Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2025-07-17T08:42:44.0854911Z From https://github.com/pytorch/pytorch
2025-07-17T08:42:44.0855543Z  - [deleted]                 (none)     -> ciflow/inductor/158527
2025-07-17T08:42:44.1818551Z  - [deleted]                 (none)     -> ciflow/rocm/158037
2025-07-17T08:42:44.1820232Z  - [deleted]                 (none)     -> ciflow/trunk/158037
2025-07-17T08:42:44.1822297Z  - [deleted]                 (none)     -> ciflow/trunk/158416
2025-07-17T08:42:44.1824273Z  - [deleted]                 (none)     -> ciflow/trunk/158453
2025-07-17T08:42:44.1826191Z  - [deleted]                 (none)     -> ciflow/trunk/158489
2025-07-17T08:42:44.1827864Z  - [deleted]                 (none)     -> ciflow/trunk/158527
2025-07-17T08:42:46.9348043Z    2050a831324..a9ab31ece73  gh/Sidharth123-cpu/43/base -> origin/gh/Sidharth123-cpu/43/base
2025-07-17T08:42:46.9350119Z    ae36a219aec..bc020792a93  gh/Sidharth123-cpu/43/head -> origin/gh/Sidharth123-cpu/43/head
2025-07-17T08:42:46.9356921Z  + ff80f13c50a...558aa0377ea gh/Sidharth123-cpu/43/orig -> origin/gh/Sidharth123-cpu/43/orig  (forced update)
2025-07-17T08:42:46.9372021Z    a0dbea65e6f..c95331cd836  gh/XuehaiPan/179/base   -> origin/gh/XuehaiPan/179/base
2025-07-17T08:42:46.9375111Z    1efebd56387..a83ce9d5fe0  gh/XuehaiPan/179/head   -> origin/gh/XuehaiPan/179/head
2025-07-17T08:42:46.9378724Z  + 160d337cf57...aba7ddb1246 gh/XuehaiPan/179/orig   -> origin/gh/XuehaiPan/179/orig  (forced update)
2025-07-17T08:42:46.9383172Z    f589cb4a722..8e2c6ff709f  gh/XuehaiPan/249/base   -> origin/gh/XuehaiPan/249/base
2025-07-17T08:42:46.9386834Z    3e66bd8fa8d..d05480c2360  gh/XuehaiPan/249/head   -> origin/gh/XuehaiPan/249/head
2025-07-17T08:42:46.9390935Z  + 2f4c9e4605a...3f981074efb gh/XuehaiPan/249/orig   -> origin/gh/XuehaiPan/249/orig  (forced update)
2025-07-17T08:42:46.9393780Z    b7e61f8ef3e..018d000fae4  gh/XuehaiPan/253/base   -> origin/gh/XuehaiPan/253/base
2025-07-17T08:42:46.9397169Z    72fa39125dc..80a7d88057d  gh/XuehaiPan/253/head   -> origin/gh/XuehaiPan/253/head
2025-07-17T08:42:46.9400229Z  + b46f356160b...21a43a7bbae gh/XuehaiPan/253/orig   -> origin/gh/XuehaiPan/253/orig  (forced update)
2025-07-17T08:42:46.9403839Z    e0941789541..7c0318f29ec  gh/XuehaiPan/271/base   -> origin/gh/XuehaiPan/271/base
2025-07-17T08:42:46.9406941Z    86295317601..870921f43d6  gh/XuehaiPan/271/head   -> origin/gh/XuehaiPan/271/head
2025-07-17T08:42:46.9410298Z  + 9a306d0fb65...68cc42ef0d8 gh/XuehaiPan/271/orig   -> origin/gh/XuehaiPan/271/orig  (forced update)
2025-07-17T08:42:46.9415839Z    7ca1c2847f5..dcc1adf677c  gh/XuehaiPan/290/base   -> origin/gh/XuehaiPan/290/base
2025-07-17T08:42:46.9418814Z    98d8e32c5ff..732eaba3a4c  gh/XuehaiPan/290/head   -> origin/gh/XuehaiPan/290/head
2025-07-17T08:42:46.9422097Z  + 5c17d8b6786...6fb43653468 gh/XuehaiPan/290/orig   -> origin/gh/XuehaiPan/290/orig  (forced update)
2025-07-17T08:42:46.9429836Z    e53d3c12403..1709d95d9b7  gh/XuehaiPan/328/base   -> origin/gh/XuehaiPan/328/base
2025-07-17T08:42:46.9432718Z    778e4c0f6c0..5505f95fff8  gh/XuehaiPan/328/head   -> origin/gh/XuehaiPan/328/head
2025-07-17T08:42:46.9435684Z  + be51f9b499d...61f4c7a0b85 gh/XuehaiPan/328/orig   -> origin/gh/XuehaiPan/328/orig  (forced update)
2025-07-17T08:42:46.9441042Z    954df72965f..ff39e21651d  gh/XuehaiPan/343/base   -> origin/gh/XuehaiPan/343/base
2025-07-17T08:42:46.9443671Z    208110970fc..aaeb58b661f  gh/XuehaiPan/343/head   -> origin/gh/XuehaiPan/343/head
2025-07-17T08:42:46.9446622Z  + 2b719d36b2b...b8b4d96046c gh/XuehaiPan/343/orig   -> origin/gh/XuehaiPan/343/orig  (forced update)
2025-07-17T08:42:46.9449543Z    300ccaee7d6..2dbed576805  gh/XuehaiPan/344/base   -> origin/gh/XuehaiPan/344/base
2025-07-17T08:42:46.9452405Z    e295d4dde43..6d2a855c305  gh/XuehaiPan/344/head   -> origin/gh/XuehaiPan/344/head
2025-07-17T08:42:46.9455573Z  + f935d46b495...4ab1f01cf77 gh/XuehaiPan/344/orig   -> origin/gh/XuehaiPan/344/orig  (forced update)
2025-07-17T08:42:46.9458466Z    b7cadd195cd..c3ba852a5bb  gh/XuehaiPan/345/base   -> origin/gh/XuehaiPan/345/base
2025-07-17T08:42:46.9461180Z    0418cbcd8bf..73f3202632f  gh/XuehaiPan/345/head   -> origin/gh/XuehaiPan/345/head
2025-07-17T08:42:46.9464067Z  + 8546dffb863...571fcf292cf gh/XuehaiPan/345/orig   -> origin/gh/XuehaiPan/345/orig  (forced update)
2025-07-17T08:42:46.9466876Z    9b227073729..bca6306c575  gh/XuehaiPan/346/base   -> origin/gh/XuehaiPan/346/base
2025-07-17T08:42:46.9469707Z    643d6117256..4a16426ad35  gh/XuehaiPan/346/head   -> origin/gh/XuehaiPan/346/head
2025-07-17T08:42:46.9472470Z  + 89d9ad2db0b...add1f8e950a gh/XuehaiPan/346/orig   -> origin/gh/XuehaiPan/346/orig  (forced update)
2025-07-17T08:42:46.9475462Z    489125cb6cc..f3d99ebd3ff  gh/XuehaiPan/347/base   -> origin/gh/XuehaiPan/347/base
2025-07-17T08:42:46.9478403Z    ebba3522a23..412b8264451  gh/XuehaiPan/347/head   -> origin/gh/XuehaiPan/347/head
2025-07-17T08:42:46.9481328Z  + 09401e0e782...363b8bf2b22 gh/XuehaiPan/347/orig   -> origin/gh/XuehaiPan/347/orig  (forced update)
2025-07-17T08:42:46.9484095Z    089ec8d7805..f6491b39f7f  gh/XuehaiPan/348/base   -> origin/gh/XuehaiPan/348/base
2025-07-17T08:42:46.9486908Z    e5a3b6d6286..38a8ad83f44  gh/XuehaiPan/348/head   -> origin/gh/XuehaiPan/348/head
2025-07-17T08:42:46.9489780Z  + 2c3812408a2...8877b8ebfff gh/XuehaiPan/348/orig   -> origin/gh/XuehaiPan/348/orig  (forced update)
2025-07-17T08:42:46.9492829Z    2a15541b5dd..67f00597263  gh/XuehaiPan/350/base   -> origin/gh/XuehaiPan/350/base
2025-07-17T08:42:46.9495429Z    16875b228be..55473096e2a  gh/XuehaiPan/350/head   -> origin/gh/XuehaiPan/350/head
2025-07-17T08:42:46.9498213Z  + dc278149c83...21623fdef59 gh/XuehaiPan/350/orig   -> origin/gh/XuehaiPan/350/orig  (forced update)
2025-07-17T08:42:46.9501341Z    ad1ac3ef58b..71b54e5404c  gh/XuehaiPan/352/base   -> origin/gh/XuehaiPan/352/base
2025-07-17T08:42:46.9504219Z    9e83151d858..1be02042962  gh/XuehaiPan/352/head   -> origin/gh/XuehaiPan/352/head
2025-07-17T08:42:46.9507046Z  + 82ca4b4d10b...6e103440bc9 gh/XuehaiPan/352/orig   -> origin/gh/XuehaiPan/352/orig  (forced update)
2025-07-17T08:42:46.9510746Z    f6c8169abca..a8fb7839752  gh/XuehaiPan/365/base   -> origin/gh/XuehaiPan/365/base
2025-07-17T08:42:46.9513464Z    845a8df6b05..0ae2a3337b7  gh/XuehaiPan/365/head   -> origin/gh/XuehaiPan/365/head
2025-07-17T08:42:46.9515965Z  + 6d02d712728...3fe78eb180a gh/XuehaiPan/365/orig   -> origin/gh/XuehaiPan/365/orig  (forced update)
2025-07-17T08:42:46.9585794Z  * [new branch]              gh/guangyey/172/base    -> origin/gh/guangyey/172/base
2025-07-17T08:42:46.9588142Z  * [new branch]              gh/guangyey/172/head    -> origin/gh/guangyey/172/head
2025-07-17T08:42:46.9589288Z  * [new branch]              gh/guangyey/172/orig    -> origin/gh/guangyey/172/orig
2025-07-17T08:42:46.9633194Z    6f59bfa1427..17d1d9b3770  gh/lw/2/head            -> origin/gh/lw/2/head
2025-07-17T08:42:46.9635881Z  + e1c3489f56e...4ec6221db18 gh/lw/2/orig            -> origin/gh/lw/2/orig  (forced update)
2025-07-17T08:42:46.9651293Z    ed051c30846..1cc0eff541c  gh/oulgen/44/base       -> origin/gh/oulgen/44/base
2025-07-17T08:42:46.9654570Z    f5ffcbfbc36..6c2f5866ffa  gh/oulgen/44/head       -> origin/gh/oulgen/44/head
2025-07-17T08:42:46.9657427Z  + e117f879a40...d27686412a1 gh/oulgen/44/orig       -> origin/gh/oulgen/44/orig  (forced update)
2025-07-17T08:42:46.9706414Z    d9426a81d2a..39ac189808c  main                    -> origin/main
2025-07-17T08:42:46.9714043Z  + cb80bda2db4...d0b8cc6b79c mlazos/hop-modes        -> origin/mlazos/hop-modes  (forced update)
2025-07-17T08:42:46.9717070Z  + 3c611c95e3a...228824d2cbe mlazos/nested-dc        -> origin/mlazos/nested-dc  (forced update)
2025-07-17T08:42:46.9722452Z    ab43fe4bdf5..9e5df57ebb4  nightly                 -> origin/nightly
2025-07-17T08:42:46.9736637Z  + 3133820e94e...67b6d88e85c update_submodule_FBGEMM -> origin/update_submodule_FBGEMM  (forced update)
2025-07-17T08:42:46.9739901Z  + a8647ccd0db...aa6f3f622b4 update_submodule_kineto -> origin/update_submodule_kineto  (forced update)
2025-07-17T08:42:46.9742238Z    2179afd7149..2ad5c25cfc6  viable/strict           -> origin/viable/strict
2025-07-17T08:42:46.9746900Z  t [tag update]              ciflow/binaries/156049  -> ciflow/binaries/156049
2025-07-17T08:42:46.9748333Z  t [tag update]              ciflow/binaries/157432  -> ciflow/binaries/157432
2025-07-17T08:42:46.9750007Z  t [tag update]              ciflow/binaries/158104  -> ciflow/binaries/158104
2025-07-17T08:42:46.9751736Z  t [tag update]              ciflow/binaries_libtorch/156049 -> ciflow/binaries_libtorch/156049
2025-07-17T08:42:46.9752729Z  t [tag update]              ciflow/binaries_libtorch/157432 -> ciflow/binaries_libtorch/157432
2025-07-17T08:42:46.9754424Z  t [tag update]              ciflow/binaries_wheel/156049 -> ciflow/binaries_wheel/156049
2025-07-17T08:42:46.9755860Z  t [tag update]              ciflow/binaries_wheel/157432 -> ciflow/binaries_wheel/157432
2025-07-17T08:42:46.9757842Z  t [tag update]              ciflow/h100-distributed/156605 -> ciflow/h100-distributed/156605
2025-07-17T08:42:46.9759426Z  t [tag update]              ciflow/h100-distributed/156703 -> ciflow/h100-distributed/156703
2025-07-17T08:42:46.9763056Z  t [tag update]              ciflow/inductor/137400  -> ciflow/inductor/137400
2025-07-17T08:42:46.9764872Z  t [tag update]              ciflow/inductor/148180  -> ciflow/inductor/148180
2025-07-17T08:42:46.9766437Z  t [tag update]              ciflow/inductor/148328  -> ciflow/inductor/148328
2025-07-17T08:42:46.9768362Z  t [tag update]              ciflow/inductor/152624  -> ciflow/inductor/152624
2025-07-17T08:42:46.9770335Z  t [tag update]              ciflow/inductor/155452  -> ciflow/inductor/155452
2025-07-17T08:42:46.9772022Z  t [tag update]              ciflow/inductor/156049  -> ciflow/inductor/156049
2025-07-17T08:42:46.9774500Z  t [tag update]              ciflow/inductor/156605  -> ciflow/inductor/156605
2025-07-17T08:42:46.9776815Z  t [tag update]              ciflow/inductor/157635  -> ciflow/inductor/157635
2025-07-17T08:42:46.9779861Z  t [tag update]              ciflow/inductor/157993  -> ciflow/inductor/157993
2025-07-17T08:42:46.9782026Z  t [tag update]              ciflow/inductor/158072  -> ciflow/inductor/158072
2025-07-17T08:42:46.9783608Z  t [tag update]              ciflow/inductor/158104  -> ciflow/inductor/158104
2025-07-17T08:42:46.9787400Z  t [tag update]              ciflow/inductor/158430  -> ciflow/inductor/158430
2025-07-17T08:42:46.9789211Z  t [tag update]              ciflow/inductor/158460  -> ciflow/inductor/158460
2025-07-17T08:42:46.9791968Z  t [tag update]              ciflow/inductor/158535  -> ciflow/inductor/158535
2025-07-17T08:42:46.9793390Z  * [new tag]                 ciflow/inductor/158543  -> ciflow/inductor/158543
2025-07-17T08:42:46.9795352Z  t [tag update]              ciflow/mps/157553       -> ciflow/mps/157553
2025-07-17T08:42:46.9798830Z  t [tag update]              ciflow/trunk/137400     -> ciflow/trunk/137400
2025-07-17T08:42:46.9800630Z  t [tag update]              ciflow/trunk/148180     -> ciflow/trunk/148180
2025-07-17T08:42:46.9802097Z  t [tag update]              ciflow/trunk/148328     -> ciflow/trunk/148328
2025-07-17T08:42:46.9803858Z  t [tag update]              ciflow/trunk/152624     -> ciflow/trunk/152624
2025-07-17T08:42:46.9805776Z  t [tag update]              ciflow/trunk/156049     -> ciflow/trunk/156049
2025-07-17T08:42:46.9807438Z  t [tag update]              ciflow/trunk/156605     -> ciflow/trunk/156605
2025-07-17T08:42:46.9809463Z  t [tag update]              ciflow/trunk/157432     -> ciflow/trunk/157432
2025-07-17T08:42:46.9810990Z  t [tag update]              ciflow/trunk/157550     -> ciflow/trunk/157550
2025-07-17T08:42:46.9812468Z  t [tag update]              ciflow/trunk/157552     -> ciflow/trunk/157552
2025-07-17T08:42:46.9814723Z  t [tag update]              ciflow/trunk/158072     -> ciflow/trunk/158072
2025-07-17T08:42:46.9816306Z  t [tag update]              ciflow/trunk/158104     -> ciflow/trunk/158104
2025-07-17T08:42:46.9818893Z  t [tag update]              ciflow/trunk/158430     -> ciflow/trunk/158430
2025-07-17T08:42:46.9820278Z  * [new tag]                 ciflow/trunk/158541     -> ciflow/trunk/158541
2025-07-17T08:42:46.9821455Z  * [new tag]                 ciflow/trunk/158543     -> ciflow/trunk/158543
2025-07-17T08:42:46.9823780Z  t [tag update]              ciflow/xpu/158533       -> ciflow/xpu/158533
2025-07-17T08:42:46.9824947Z  * [new tag]                 ciflow/xpu/158542       -> ciflow/xpu/158542
2025-07-17T08:42:46.9829114Z  * [new tag]                 trunk/04349f9ee541c7d07cc057bbe739f46bd4c30dcc -> trunk/04349f9ee541c7d07cc057bbe739f46bd4c30dcc
2025-07-17T08:42:46.9834484Z  * [new tag]                 trunk/09db3a22e8783c4841697317688ba9467c7cc457 -> trunk/09db3a22e8783c4841697317688ba9467c7cc457
2025-07-17T08:42:46.9867974Z  * [new tag]                 trunk/39ac189808c61588f3594dbc2fc1d69bb6194c47 -> trunk/39ac189808c61588f3594dbc2fc1d69bb6194c47
2025-07-17T08:42:46.9925446Z  * [new tag]                 trunk/9636e2cfd3e995ef977f670ad47e8e895296d992 -> trunk/9636e2cfd3e995ef977f670ad47e8e895296d992
2025-07-17T08:42:46.9931747Z  * [new tag]                 trunk/9f37cce69334bccebf4b21503f0047d0c0bb320c -> trunk/9f37cce69334bccebf4b21503f0047d0c0bb320c
2025-07-17T08:42:46.9935752Z  * [new tag]                 trunk/a38f433be2e94a64b095a44ba39879d02d0c2316 -> trunk/a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:42:46.9969411Z  * [new tag]                 trunk/d76323d41742cbc05ec6857319b267d2c7ea8fd9 -> trunk/d76323d41742cbc05ec6857319b267d2c7ea8fd9
2025-07-17T08:42:47.1379485Z [command]/usr/bin/git rev-parse --verify --quiet a38f433be2e94a64b095a44ba39879d02d0c2316^{object}
2025-07-17T08:42:47.1472256Z a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:42:47.1479822Z ##[endgroup]
2025-07-17T08:42:47.1481398Z ##[group]Determining the checkout info
2025-07-17T08:42:47.1482301Z ##[endgroup]
2025-07-17T08:42:47.1486508Z [command]/usr/bin/git sparse-checkout disable
2025-07-17T08:42:47.1707664Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2025-07-17T08:42:47.1747548Z ##[group]Checking out the ref
2025-07-17T08:42:47.1750857Z [command]/usr/bin/git checkout --progress --force a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:42:47.6299529Z Previous HEAD position was 7770f763058 Update on "[hop] set capture_scalar_outputs=True by default for compiled hops"
2025-07-17T08:42:47.6320066Z HEAD is now at a38f433be2e [Docker builds] Move from Miniconda to Miniforge (#158370)
2025-07-17T08:42:47.6499718Z ##[endgroup]
2025-07-17T08:42:47.6500308Z ##[group]Setting up auth for fetching submodules
2025-07-17T08:42:47.6506023Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-07-17T08:42:47.6553774Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2025-07-17T08:42:47.6591671Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2025-07-17T08:42:47.6633425Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2025-07-17T08:42:47.6668834Z ##[endgroup]
2025-07-17T08:42:47.6669408Z ##[group]Fetching submodules
2025-07-17T08:42:47.6671873Z [command]/usr/bin/git submodule sync --recursive
2025-07-17T08:42:47.7063649Z Synchronizing submodule url for 'android/libs/fbjni'
2025-07-17T08:42:47.7124041Z Synchronizing submodule url for 'third_party/FP16'
2025-07-17T08:42:47.7182310Z Synchronizing submodule url for 'third_party/FXdiv'
2025-07-17T08:42:47.7242927Z Synchronizing submodule url for 'third_party/NNPACK'
2025-07-17T08:42:47.7303762Z Synchronizing submodule url for 'third_party/NVTX'
2025-07-17T08:42:47.7362389Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator'
2025-07-17T08:42:47.7423118Z Synchronizing submodule url for 'third_party/XNNPACK'
2025-07-17T08:42:47.7501878Z Synchronizing submodule url for 'third_party/aiter'
2025-07-17T08:42:47.7556009Z Synchronizing submodule url for 'third_party/aiter/3rdparty/composable_kernel'
2025-07-17T08:42:47.7625526Z Synchronizing submodule url for 'third_party/benchmark'
2025-07-17T08:42:47.7683791Z Synchronizing submodule url for 'third_party/composable_kernel'
2025-07-17T08:42:47.7747049Z Synchronizing submodule url for 'third_party/cpp-httplib'
2025-07-17T08:42:47.7803513Z Synchronizing submodule url for 'third_party/cpuinfo'
2025-07-17T08:42:47.7862859Z Synchronizing submodule url for 'third_party/cudnn_frontend'
2025-07-17T08:42:47.7918563Z Synchronizing submodule url for 'third_party/cutlass'
2025-07-17T08:42:47.7992420Z Synchronizing submodule url for 'third_party/fbgemm'
2025-07-17T08:42:47.8049235Z Synchronizing submodule url for 'third_party/fbgemm/external/asmjit'
2025-07-17T08:42:47.8103163Z Synchronizing submodule url for 'third_party/fbgemm/external/composable_kernel'
2025-07-17T08:42:47.8163074Z Synchronizing submodule url for 'third_party/fbgemm/external/cpuinfo'
2025-07-17T08:42:47.8224450Z Synchronizing submodule url for 'third_party/fbgemm/external/cutlass'
2025-07-17T08:42:47.8286519Z Synchronizing submodule url for 'third_party/fbgemm/external/googletest'
2025-07-17T08:42:47.8341639Z Synchronizing submodule url for 'third_party/fbgemm/external/hipify_torch'
2025-07-17T08:42:47.8393289Z Synchronizing submodule url for 'third_party/fbgemm/external/json'
2025-07-17T08:42:47.8456475Z Synchronizing submodule url for 'third_party/flash-attention'
2025-07-17T08:42:47.8511959Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel'
2025-07-17T08:42:47.8576147Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass'
2025-07-17T08:42:47.8651324Z Synchronizing submodule url for 'third_party/flatbuffers'
2025-07-17T08:42:47.8717593Z Synchronizing submodule url for 'third_party/fmt'
2025-07-17T08:42:47.8772454Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp'
2025-07-17T08:42:47.8834900Z Synchronizing submodule url for 'third_party/gloo'
2025-07-17T08:42:47.8892566Z Synchronizing submodule url for 'third_party/googletest'
2025-07-17T08:42:47.8953472Z Synchronizing submodule url for 'third_party/ideep'
2025-07-17T08:42:47.9006671Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn'
2025-07-17T08:42:47.9070639Z Synchronizing submodule url for 'third_party/ittapi'
2025-07-17T08:42:47.9130451Z Synchronizing submodule url for 'third_party/kineto'
2025-07-17T08:42:47.9192111Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-17T08:42:47.9242807Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-17T08:42:47.9296112Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-17T08:42:47.9353672Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-17T08:42:47.9405597Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-17T08:42:47.9452622Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-17T08:42:47.9510806Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-17T08:42:47.9564867Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-17T08:42:47.9623909Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-17T08:42:47.9677705Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-17T08:42:47.9732211Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt'
2025-07-17T08:42:47.9788121Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest'
2025-07-17T08:42:47.9846101Z Synchronizing submodule url for 'third_party/kleidiai'
2025-07-17T08:42:47.9909147Z Synchronizing submodule url for 'third_party/mimalloc'
2025-07-17T08:42:47.9966900Z Synchronizing submodule url for 'third_party/nlohmann'
2025-07-17T08:42:48.0028997Z Synchronizing submodule url for 'third_party/onnx'
2025-07-17T08:42:48.0103393Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11'
2025-07-17T08:42:48.0168864Z Synchronizing submodule url for 'third_party/opentelemetry-cpp'
2025-07-17T08:42:48.0226490Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-17T08:42:48.0276655Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-17T08:42:48.0329997Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-17T08:42:48.0386428Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-17T08:42:48.0438657Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-17T08:42:48.0491339Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-17T08:42:48.0550334Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-17T08:42:48.0596340Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-17T08:42:48.0650083Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-17T08:42:48.0712394Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-17T08:42:48.0795310Z Synchronizing submodule url for 'third_party/pocketfft'
2025-07-17T08:42:48.0857204Z Synchronizing submodule url for 'third_party/protobuf'
2025-07-17T08:42:48.0912906Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark'
2025-07-17T08:42:48.0965303Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest'
2025-07-17T08:42:48.1025274Z Synchronizing submodule url for 'third_party/psimd'
2025-07-17T08:42:48.1086214Z Synchronizing submodule url for 'third_party/pthreadpool'
2025-07-17T08:42:48.1149232Z Synchronizing submodule url for 'third_party/pybind11'
2025-07-17T08:42:48.1207437Z Synchronizing submodule url for 'third_party/python-peachpy'
2025-07-17T08:42:48.1267976Z Synchronizing submodule url for 'third_party/sleef'
2025-07-17T08:42:48.1327674Z Synchronizing submodule url for 'third_party/tensorpipe'
2025-07-17T08:42:48.1386333Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest'
2025-07-17T08:42:48.1439836Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop'
2025-07-17T08:42:48.1498185Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv'
2025-07-17T08:42:48.1549454Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11'
2025-07-17T08:42:48.1600220Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-17T08:42:48.1690451Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2025-07-17T08:42:48.2392524Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2025-07-17T08:42:48.2741056Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2025-07-17T08:42:48.3095786Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2025-07-17T08:42:48.3450541Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2025-07-17T08:42:48.3828339Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07'
2025-07-17T08:42:48.4187202Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1'
2025-07-17T08:42:48.4756210Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883'
2025-07-17T08:42:48.5252012Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150'
2025-07-17T08:42:48.5778458Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf'
2025-07-17T08:42:48.6186195Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f'
2025-07-17T08:42:48.6755198Z Submodule path 'third_party/composable_kernel': checked out '434d19f696da62c12b5372b32cbc9ba968588d7e'
2025-07-17T08:42:48.7147887Z Submodule path 'third_party/cpp-httplib': checked out '3af7f2c16147f3fbc6e4d717032daf505dc1652c'
2025-07-17T08:42:48.7517129Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d'
2025-07-17T08:42:48.7913289Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6'
2025-07-17T08:42:48.8390955Z Submodule path 'third_party/cutlass': checked out 'b995f933179c22d3fe0d871c3a53d11e4681950f'
2025-07-17T08:42:48.8887403Z Submodule path 'third_party/fbgemm': checked out '157e88b750c452bef2ab4653fe9d1eeb151ce4c3'
2025-07-17T08:42:48.9233736Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'e5d7c0bd5d9aec44d68830187138149e6a8c4e32'
2025-07-17T08:42:48.9638785Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '4a61bdd4bd4ed730e078aebc7c0fcf046ff29406'
2025-07-17T08:42:49.0005810Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349'
2025-07-17T08:42:49.0432488Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '3ed8d2ec4ba35ef5d9d8353826209b6f868f63d3'
2025-07-17T08:42:49.0790168Z Submodule path 'third_party/fbgemm/external/googletest': checked out 'f8d7d77c06936315286eb55f8de22cd23c188571'
2025-07-17T08:42:49.1111882Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out 'a4337c69fe0e2552a7b7b0669178926beeed828c'
2025-07-17T08:42:49.1493105Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03'
2025-07-17T08:42:49.1912234Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5'
2025-07-17T08:42:49.2421796Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33'
2025-07-17T08:42:49.2862952Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420'
2025-07-17T08:42:49.3306759Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757'
2025-07-17T08:42:49.3697393Z Submodule path 'third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21'
2025-07-17T08:42:49.4058631Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2025-07-17T08:42:49.4448436Z Submodule path 'third_party/gloo': checked out 'c7b7b022c124d9643957d9bd55f57ac59fce8fa2'
2025-07-17T08:42:49.4809011Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-07-17T08:42:49.5189283Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3'
2025-07-17T08:42:49.5712735Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d'
2025-07-17T08:42:49.6105743Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959'
2025-07-17T08:42:49.6491676Z Submodule path 'third_party/kineto': checked out '5e7501833f1021ce6f618572d3baf657b6319658'
2025-07-17T08:42:49.6853000Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2025-07-17T08:42:49.7208942Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2025-07-17T08:42:49.7560883Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2025-07-17T08:42:49.7905703Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2025-07-17T08:42:49.8246567Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2025-07-17T08:42:49.8577064Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2025-07-17T08:42:49.8916218Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2025-07-17T08:42:49.9264855Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2025-07-17T08:42:49.9658708Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2025-07-17T08:42:49.9988063Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2025-07-17T08:42:50.0336065Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2025-07-17T08:42:50.0670289Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2025-07-17T08:42:50.1051700Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7'
2025-07-17T08:42:50.1454167Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e'
2025-07-17T08:42:50.1880783Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72'
2025-07-17T08:42:50.2445575Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83'
2025-07-17T08:42:50.2865375Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-07-17T08:42:50.3308751Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2025-07-17T08:42:50.3652813Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2025-07-17T08:42:50.4001254Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2025-07-17T08:42:50.4335026Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2025-07-17T08:42:50.4730970Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2025-07-17T08:42:50.5073845Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2025-07-17T08:42:50.5400364Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2025-07-17T08:42:50.5738192Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2025-07-17T08:42:50.6099465Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2025-07-17T08:42:50.6456069Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-07-17T08:42:50.6989657Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2025-07-17T08:42:50.7381744Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa'
2025-07-17T08:42:50.7967603Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2025-07-17T08:42:50.8300263Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2025-07-17T08:42:50.8628202Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2025-07-17T08:42:50.8986922Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2025-07-17T08:42:50.9328986Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2025-07-17T08:42:50.9726337Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-07-17T08:42:51.0080594Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2025-07-17T08:42:51.0433024Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68'
2025-07-17T08:42:51.0797676Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2025-07-17T08:42:51.1113747Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2025-07-17T08:42:51.1425893Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2025-07-17T08:42:51.1916478Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2025-07-17T08:42:51.2271314Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2025-07-17T08:42:51.2571417Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2025-07-17T08:42:51.2711456Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2025-07-17T08:42:51.3086707Z Entering 'android/libs/fbjni'
2025-07-17T08:42:51.3155450Z Entering 'third_party/FP16'
2025-07-17T08:42:51.3224041Z Entering 'third_party/FXdiv'
2025-07-17T08:42:51.3283079Z Entering 'third_party/NNPACK'
2025-07-17T08:42:51.3347526Z Entering 'third_party/NVTX'
2025-07-17T08:42:51.3415212Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-17T08:42:51.3481345Z Entering 'third_party/XNNPACK'
2025-07-17T08:42:51.3566045Z Entering 'third_party/aiter'
2025-07-17T08:42:51.3630663Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-17T08:42:51.3706420Z Entering 'third_party/benchmark'
2025-07-17T08:42:51.3768865Z Entering 'third_party/composable_kernel'
2025-07-17T08:42:51.3845447Z Entering 'third_party/cpp-httplib'
2025-07-17T08:42:51.3910998Z Entering 'third_party/cpuinfo'
2025-07-17T08:42:51.3976502Z Entering 'third_party/cudnn_frontend'
2025-07-17T08:42:51.4041009Z Entering 'third_party/cutlass'
2025-07-17T08:42:51.4118505Z Entering 'third_party/fbgemm'
2025-07-17T08:42:51.4196585Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-17T08:42:51.4256334Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-17T08:42:51.4324828Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-17T08:42:51.4388069Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-17T08:42:51.4453379Z Entering 'third_party/fbgemm/external/googletest'
2025-07-17T08:42:51.4510703Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-17T08:42:51.4569046Z Entering 'third_party/fbgemm/external/json'
2025-07-17T08:42:51.4631666Z Entering 'third_party/flash-attention'
2025-07-17T08:42:51.4685598Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-17T08:42:51.4752097Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-17T08:42:51.4825499Z Entering 'third_party/flatbuffers'
2025-07-17T08:42:51.4887211Z Entering 'third_party/fmt'
2025-07-17T08:42:51.4951310Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-17T08:42:51.5018677Z Entering 'third_party/gloo'
2025-07-17T08:42:51.5083020Z Entering 'third_party/googletest'
2025-07-17T08:42:51.5149899Z Entering 'third_party/ideep'
2025-07-17T08:42:51.5211707Z Entering 'third_party/ideep/mkl-dnn'
2025-07-17T08:42:51.5285593Z Entering 'third_party/ittapi'
2025-07-17T08:42:51.5349445Z Entering 'third_party/kineto'
2025-07-17T08:42:51.5410589Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-17T08:42:51.5468038Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-17T08:42:51.5529236Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-17T08:42:51.5589726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-17T08:42:51.5646876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-17T08:42:51.5701332Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-17T08:42:51.5765736Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-17T08:42:51.5830464Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-17T08:42:51.5885897Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-17T08:42:51.5950864Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-17T08:42:51.6012897Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-17T08:42:51.6072964Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-17T08:42:51.6136037Z Entering 'third_party/kleidiai'
2025-07-17T08:42:51.6200409Z Entering 'third_party/mimalloc'
2025-07-17T08:42:51.6271422Z Entering 'third_party/nlohmann'
2025-07-17T08:42:51.6341229Z Entering 'third_party/onnx'
2025-07-17T08:42:51.6425235Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-17T08:42:51.6493981Z Entering 'third_party/opentelemetry-cpp'
2025-07-17T08:42:51.6554869Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-17T08:42:51.6613166Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-17T08:42:51.6672212Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-17T08:42:51.6726260Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-17T08:42:51.6785405Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-17T08:42:51.6844405Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-17T08:42:51.6894278Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-17T08:42:51.6951320Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-17T08:42:51.7010296Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-17T08:42:51.7076345Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-17T08:42:51.7155123Z Entering 'third_party/pocketfft'
2025-07-17T08:42:51.7221185Z Entering 'third_party/protobuf'
2025-07-17T08:42:51.7288729Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-17T08:42:51.7352597Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-17T08:42:51.7417757Z Entering 'third_party/psimd'
2025-07-17T08:42:51.7483161Z Entering 'third_party/pthreadpool'
2025-07-17T08:42:51.7554179Z Entering 'third_party/pybind11'
2025-07-17T08:42:51.7615745Z Entering 'third_party/python-peachpy'
2025-07-17T08:42:51.7676328Z Entering 'third_party/sleef'
2025-07-17T08:42:51.7743493Z Entering 'third_party/tensorpipe'
2025-07-17T08:42:51.7804459Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-17T08:42:51.7862470Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-17T08:42:51.7916587Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-17T08:42:51.7976936Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-17T08:42:51.8031190Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-17T08:42:51.8119884Z ##[endgroup]
2025-07-17T08:42:51.8120588Z ##[group]Persisting credentials for submodules
2025-07-17T08:42:51.8126168Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2025-07-17T08:42:51.8501963Z Entering 'android/libs/fbjni'
2025-07-17T08:42:51.8541097Z url.https://github.com/.insteadof
2025-07-17T08:42:51.8541686Z url.https://github.com/.insteadof
2025-07-17T08:42:51.8591149Z Entering 'third_party/FP16'
2025-07-17T08:42:51.8632196Z url.https://github.com/.insteadof
2025-07-17T08:42:51.8632726Z url.https://github.com/.insteadof
2025-07-17T08:42:51.8685808Z Entering 'third_party/FXdiv'
2025-07-17T08:42:51.8720485Z url.https://github.com/.insteadof
2025-07-17T08:42:51.8720921Z url.https://github.com/.insteadof
2025-07-17T08:42:51.8773948Z Entering 'third_party/NNPACK'
2025-07-17T08:42:51.8813522Z url.https://github.com/.insteadof
2025-07-17T08:42:51.8813963Z url.https://github.com/.insteadof
2025-07-17T08:42:51.8867988Z Entering 'third_party/NVTX'
2025-07-17T08:42:51.8911700Z url.https://github.com/.insteadof
2025-07-17T08:42:51.8912222Z url.https://github.com/.insteadof
2025-07-17T08:42:51.8964741Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-17T08:42:51.9001232Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9001652Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9055583Z Entering 'third_party/XNNPACK'
2025-07-17T08:42:51.9093362Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9093788Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9167046Z Entering 'third_party/aiter'
2025-07-17T08:42:51.9207595Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9208020Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9264280Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-17T08:42:51.9299598Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9300038Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9363022Z Entering 'third_party/benchmark'
2025-07-17T08:42:51.9399388Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9400191Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9455031Z Entering 'third_party/composable_kernel'
2025-07-17T08:42:51.9493692Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9494117Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9560504Z Entering 'third_party/cpp-httplib'
2025-07-17T08:42:51.9596469Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9596979Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9646596Z Entering 'third_party/cpuinfo'
2025-07-17T08:42:51.9681881Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9682328Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9736289Z Entering 'third_party/cudnn_frontend'
2025-07-17T08:42:51.9775462Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9775892Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9830543Z Entering 'third_party/cutlass'
2025-07-17T08:42:51.9871375Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9871814Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9935633Z Entering 'third_party/fbgemm'
2025-07-17T08:42:51.9976199Z url.https://github.com/.insteadof
2025-07-17T08:42:51.9977019Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0032140Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-17T08:42:52.0072107Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0072428Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0124622Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-17T08:42:52.0159002Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0159410Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0217678Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-17T08:42:52.0253939Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0254374Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0309209Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-17T08:42:52.0341685Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0342151Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0401704Z Entering 'third_party/fbgemm/external/googletest'
2025-07-17T08:42:52.0437720Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0438161Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0484593Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-17T08:42:52.0521533Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0522005Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0570192Z Entering 'third_party/fbgemm/external/json'
2025-07-17T08:42:52.0603050Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0603482Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0664201Z Entering 'third_party/flash-attention'
2025-07-17T08:42:52.0700788Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0701217Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0751935Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-17T08:42:52.0793479Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0793913Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0853369Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-17T08:42:52.0892124Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0892581Z url.https://github.com/.insteadof
2025-07-17T08:42:52.0962810Z Entering 'third_party/flatbuffers'
2025-07-17T08:42:52.1001021Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1001442Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1056234Z Entering 'third_party/fmt'
2025-07-17T08:42:52.1094653Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1095121Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1148010Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-17T08:42:52.1185956Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1186479Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1238556Z Entering 'third_party/gloo'
2025-07-17T08:42:52.1278972Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1279392Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1328632Z Entering 'third_party/googletest'
2025-07-17T08:42:52.1365357Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1366118Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1420393Z Entering 'third_party/ideep'
2025-07-17T08:42:52.1462219Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1462644Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1514220Z Entering 'third_party/ideep/mkl-dnn'
2025-07-17T08:42:52.1553852Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1554166Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1623035Z Entering 'third_party/ittapi'
2025-07-17T08:42:52.1663476Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1663901Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1714919Z Entering 'third_party/kineto'
2025-07-17T08:42:52.1755965Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1756412Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1807322Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-17T08:42:52.1842992Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1843445Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1891583Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-17T08:42:52.1923689Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1924119Z url.https://github.com/.insteadof
2025-07-17T08:42:52.1976070Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-17T08:42:52.2008538Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2008989Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2062001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-17T08:42:52.2096776Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2097203Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2151073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-17T08:42:52.2188160Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2188579Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2239186Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-17T08:42:52.2277054Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2277478Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2335114Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-17T08:42:52.2368965Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2369418Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2423621Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-17T08:42:52.2457406Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2457830Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2513246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-17T08:42:52.2548549Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2549049Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2598291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-17T08:42:52.2636754Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2637182Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2695121Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-17T08:42:52.2727575Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2727993Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2776011Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-17T08:42:52.2815039Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2815327Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2871031Z Entering 'third_party/kleidiai'
2025-07-17T08:42:52.2907746Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2908194Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2958506Z Entering 'third_party/mimalloc'
2025-07-17T08:42:52.2999186Z url.https://github.com/.insteadof
2025-07-17T08:42:52.2999712Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3051828Z Entering 'third_party/nlohmann'
2025-07-17T08:42:52.3090338Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3091154Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3145866Z Entering 'third_party/onnx'
2025-07-17T08:42:52.3181607Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3182120Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3252230Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-17T08:42:52.3286377Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3286800Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3342952Z Entering 'third_party/opentelemetry-cpp'
2025-07-17T08:42:52.3379302Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3379763Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3435565Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-17T08:42:52.3469852Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3470208Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3521206Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-17T08:42:52.3556457Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3556909Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3605764Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-17T08:42:52.3639533Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3639987Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3692667Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-17T08:42:52.3728781Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3729236Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3785090Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-17T08:42:52.3815302Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3815745Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3867101Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-17T08:42:52.3902395Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3902892Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3954922Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-17T08:42:52.3987425Z url.https://github.com/.insteadof
2025-07-17T08:42:52.3987866Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4039200Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-17T08:42:52.4075524Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4075970Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4126419Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-17T08:42:52.4160217Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4160631Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4212988Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-17T08:42:52.4250298Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4250715Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4333646Z Entering 'third_party/pocketfft'
2025-07-17T08:42:52.4374717Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4375149Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4428742Z Entering 'third_party/protobuf'
2025-07-17T08:42:52.4465006Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4465551Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4520385Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-17T08:42:52.4555726Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4556152Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4604482Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-17T08:42:52.4640053Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4640476Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4694921Z Entering 'third_party/psimd'
2025-07-17T08:42:52.4734576Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4735009Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4788396Z Entering 'third_party/pthreadpool'
2025-07-17T08:42:52.4831674Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4832103Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4884426Z Entering 'third_party/pybind11'
2025-07-17T08:42:52.4922008Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4922456Z url.https://github.com/.insteadof
2025-07-17T08:42:52.4976633Z Entering 'third_party/python-peachpy'
2025-07-17T08:42:52.5017753Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5018199Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5069984Z Entering 'third_party/sleef'
2025-07-17T08:42:52.5106438Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5106920Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5160478Z Entering 'third_party/tensorpipe'
2025-07-17T08:42:52.5195424Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5195859Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5245033Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-17T08:42:52.5282278Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5282716Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5337173Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-17T08:42:52.5371778Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5372258Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5427914Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-17T08:42:52.5462543Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5462977Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5511322Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-17T08:42:52.5550227Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5550564Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5600681Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-17T08:42:52.5638518Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5638960Z url.https://github.com/.insteadof
2025-07-17T08:42:52.5724031Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2025-07-17T08:42:52.6092022Z Entering 'android/libs/fbjni'
2025-07-17T08:42:52.6157644Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-07-17T08:42:52.6192557Z Entering 'third_party/FP16'
2025-07-17T08:42:52.6256901Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-07-17T08:42:52.6288745Z Entering 'third_party/FXdiv'
2025-07-17T08:42:52.6347501Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-07-17T08:42:52.6380368Z Entering 'third_party/NNPACK'
2025-07-17T08:42:52.6438931Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-07-17T08:42:52.6472855Z Entering 'third_party/NVTX'
2025-07-17T08:42:52.6536167Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-07-17T08:42:52.6569507Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-17T08:42:52.6626634Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-07-17T08:42:52.6655046Z Entering 'third_party/XNNPACK'
2025-07-17T08:42:52.6716410Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-07-17T08:42:52.6770587Z Entering 'third_party/aiter'
2025-07-17T08:42:52.6836165Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-07-17T08:42:52.6871860Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-17T08:42:52.6930765Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-07-17T08:42:52.6971262Z Entering 'third_party/benchmark'
2025-07-17T08:42:52.7028417Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-07-17T08:42:52.7064207Z Entering 'third_party/composable_kernel'
2025-07-17T08:42:52.7123135Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-07-17T08:42:52.7163567Z Entering 'third_party/cpp-httplib'
2025-07-17T08:42:52.7226712Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-07-17T08:42:52.7260095Z Entering 'third_party/cpuinfo'
2025-07-17T08:42:52.7320488Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-07-17T08:42:52.7352178Z Entering 'third_party/cudnn_frontend'
2025-07-17T08:42:52.7417780Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-07-17T08:42:52.7450637Z Entering 'third_party/cutlass'
2025-07-17T08:42:52.7509002Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-07-17T08:42:52.7552445Z Entering 'third_party/fbgemm'
2025-07-17T08:42:52.7618666Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-07-17T08:42:52.7651230Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-17T08:42:52.7708772Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-07-17T08:42:52.7740274Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-17T08:42:52.7799162Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-07-17T08:42:52.7836410Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-17T08:42:52.7898135Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-07-17T08:42:52.7928821Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-17T08:42:52.7988133Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-07-17T08:42:52.8025295Z Entering 'third_party/fbgemm/external/googletest'
2025-07-17T08:42:52.8083630Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-07-17T08:42:52.8114979Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-17T08:42:52.8179811Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-07-17T08:42:52.8207899Z Entering 'third_party/fbgemm/external/json'
2025-07-17T08:42:52.8267643Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-07-17T08:42:52.8301177Z Entering 'third_party/flash-attention'
2025-07-17T08:42:52.8360190Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-07-17T08:42:52.8395078Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-17T08:42:52.8448266Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-07-17T08:42:52.8486259Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-17T08:42:52.8548256Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-07-17T08:42:52.8594142Z Entering 'third_party/flatbuffers'
2025-07-17T08:42:52.8660328Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-07-17T08:42:52.8697005Z Entering 'third_party/fmt'
2025-07-17T08:42:52.8756627Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-07-17T08:42:52.8792574Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-17T08:42:52.8848059Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-07-17T08:42:52.8881481Z Entering 'third_party/gloo'
2025-07-17T08:42:52.8940746Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-07-17T08:42:52.8974103Z Entering 'third_party/googletest'
2025-07-17T08:42:52.9035434Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-07-17T08:42:52.9068642Z Entering 'third_party/ideep'
2025-07-17T08:42:52.9131840Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-07-17T08:42:52.9161430Z Entering 'third_party/ideep/mkl-dnn'
2025-07-17T08:42:52.9217896Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-07-17T08:42:52.9259680Z Entering 'third_party/ittapi'
2025-07-17T08:42:52.9316157Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-07-17T08:42:52.9350312Z Entering 'third_party/kineto'
2025-07-17T08:42:52.9412322Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-07-17T08:42:52.9442678Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-17T08:42:52.9501501Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-07-17T08:42:52.9526646Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-17T08:42:52.9583450Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-07-17T08:42:52.9611238Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-17T08:42:52.9663431Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-07-17T08:42:52.9695061Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-17T08:42:52.9754187Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-07-17T08:42:52.9786266Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-17T08:42:52.9841237Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-07-17T08:42:52.9868811Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-17T08:42:52.9934423Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-07-17T08:42:52.9969811Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-17T08:42:53.0024067Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-07-17T08:42:53.0055592Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-17T08:42:53.0115441Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-07-17T08:42:53.0147104Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-17T08:42:53.0201763Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-07-17T08:42:53.0238049Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-17T08:42:53.0301425Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-07-17T08:42:53.0332701Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-17T08:42:53.0393135Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-07-17T08:42:53.0427023Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-17T08:42:53.0476673Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-07-17T08:42:53.0512669Z Entering 'third_party/kleidiai'
2025-07-17T08:42:53.0574503Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-07-17T08:42:53.0605840Z Entering 'third_party/mimalloc'
2025-07-17T08:42:53.0663522Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-07-17T08:42:53.0697104Z Entering 'third_party/nlohmann'
2025-07-17T08:42:53.0761031Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-07-17T08:42:53.0792676Z Entering 'third_party/onnx'
2025-07-17T08:42:53.0855433Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-07-17T08:42:53.0904616Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-17T08:42:53.0961564Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-07-17T08:42:53.1001688Z Entering 'third_party/opentelemetry-cpp'
2025-07-17T08:42:53.1063773Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-07-17T08:42:53.1092715Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-17T08:42:53.1153066Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-07-17T08:42:53.1187888Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-17T08:42:53.1238111Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-07-17T08:42:53.1266182Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-17T08:42:53.1321823Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-07-17T08:42:53.1350549Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-17T08:42:53.1403146Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-07-17T08:42:53.1435072Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-17T08:42:53.1496791Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-07-17T08:42:53.1526113Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-17T08:42:53.1583674Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-07-17T08:42:53.1610380Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-17T08:42:53.1666233Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-07-17T08:42:53.1694662Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-17T08:42:53.1750188Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-07-17T08:42:53.1783021Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-17T08:42:53.1838833Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-07-17T08:42:53.1872503Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-17T08:42:53.1923502Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-07-17T08:42:53.1981938Z Entering 'third_party/pocketfft'
2025-07-17T08:42:53.2042134Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-07-17T08:42:53.2073893Z Entering 'third_party/protobuf'
2025-07-17T08:42:53.2137261Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-07-17T08:42:53.2171341Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-17T08:42:53.2227506Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-07-17T08:42:53.2254215Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-17T08:42:53.2307583Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-07-17T08:42:53.2345628Z Entering 'third_party/psimd'
2025-07-17T08:42:53.2399664Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-07-17T08:42:53.2430168Z Entering 'third_party/pthreadpool'
2025-07-17T08:42:53.2485434Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-07-17T08:42:53.2521181Z Entering 'third_party/pybind11'
2025-07-17T08:42:53.2578978Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-07-17T08:42:53.2610925Z Entering 'third_party/python-peachpy'
2025-07-17T08:42:53.2668348Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-07-17T08:42:53.2701358Z Entering 'third_party/sleef'
2025-07-17T08:42:53.2761970Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-07-17T08:42:53.2795713Z Entering 'third_party/tensorpipe'
2025-07-17T08:42:53.2858667Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-07-17T08:42:53.2888975Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-17T08:42:53.2948898Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-07-17T08:42:53.2980787Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-17T08:42:53.3039503Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-07-17T08:42:53.3068550Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-17T08:42:53.3121041Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-07-17T08:42:53.3153094Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-17T08:42:53.3206816Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-07-17T08:42:53.3238482Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-17T08:42:53.3299230Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-07-17T08:42:53.3602712Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2025-07-17T08:42:53.3981952Z Entering 'android/libs/fbjni'
2025-07-17T08:42:53.4044324Z Entering 'third_party/FP16'
2025-07-17T08:42:53.4106366Z Entering 'third_party/FXdiv'
2025-07-17T08:42:53.4173508Z Entering 'third_party/NNPACK'
2025-07-17T08:42:53.4239242Z Entering 'third_party/NVTX'
2025-07-17T08:42:53.4307361Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-17T08:42:53.4373883Z Entering 'third_party/XNNPACK'
2025-07-17T08:42:53.4458101Z Entering 'third_party/aiter'
2025-07-17T08:42:53.4521762Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-17T08:42:53.4592406Z Entering 'third_party/benchmark'
2025-07-17T08:42:53.4658872Z Entering 'third_party/composable_kernel'
2025-07-17T08:42:53.4731361Z Entering 'third_party/cpp-httplib'
2025-07-17T08:42:53.4795964Z Entering 'third_party/cpuinfo'
2025-07-17T08:42:53.4866111Z Entering 'third_party/cudnn_frontend'
2025-07-17T08:42:53.4924624Z Entering 'third_party/cutlass'
2025-07-17T08:42:53.5000662Z Entering 'third_party/fbgemm'
2025-07-17T08:42:53.5066520Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-17T08:42:53.5122111Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-17T08:42:53.5195623Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-17T08:42:53.5256021Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-17T08:42:53.5326664Z Entering 'third_party/fbgemm/external/googletest'
2025-07-17T08:42:53.5386864Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-17T08:42:53.5442879Z Entering 'third_party/fbgemm/external/json'
2025-07-17T08:42:53.5513692Z Entering 'third_party/flash-attention'
2025-07-17T08:42:53.5577156Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-17T08:42:53.5643767Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-17T08:42:53.5723284Z Entering 'third_party/flatbuffers'
2025-07-17T08:42:53.5792887Z Entering 'third_party/fmt'
2025-07-17T08:42:53.5856929Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-17T08:42:53.5922880Z Entering 'third_party/gloo'
2025-07-17T08:42:53.5989585Z Entering 'third_party/googletest'
2025-07-17T08:42:53.6052035Z Entering 'third_party/ideep'
2025-07-17T08:42:53.6115963Z Entering 'third_party/ideep/mkl-dnn'
2025-07-17T08:42:53.6182937Z Entering 'third_party/ittapi'
2025-07-17T08:42:53.6245228Z Entering 'third_party/kineto'
2025-07-17T08:42:53.6308790Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-17T08:42:53.6363104Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-17T08:42:53.6430448Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-17T08:42:53.6489096Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-17T08:42:53.6549738Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-17T08:42:53.6602802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-17T08:42:53.6667654Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-17T08:42:53.6724547Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-17T08:42:53.6783633Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-17T08:42:53.6843091Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-17T08:42:53.6912683Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-17T08:42:53.6970832Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-17T08:42:53.7033934Z Entering 'third_party/kleidiai'
2025-07-17T08:42:53.7089973Z Entering 'third_party/mimalloc'
2025-07-17T08:42:53.7152427Z Entering 'third_party/nlohmann'
2025-07-17T08:42:53.7221238Z Entering 'third_party/onnx'
2025-07-17T08:42:53.7303485Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-17T08:42:53.7372494Z Entering 'third_party/opentelemetry-cpp'
2025-07-17T08:42:53.7440598Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-17T08:42:53.7499792Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-17T08:42:53.7559466Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-17T08:42:53.7620349Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-17T08:42:53.7681982Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-17T08:42:53.7739612Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-17T08:42:53.7799781Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-17T08:42:53.7858125Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-17T08:42:53.7917090Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-17T08:42:53.7976593Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-17T08:42:53.8063797Z Entering 'third_party/pocketfft'
2025-07-17T08:42:53.8127991Z Entering 'third_party/protobuf'
2025-07-17T08:42:53.8195435Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-17T08:42:53.8254270Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-17T08:42:53.8318751Z Entering 'third_party/psimd'
2025-07-17T08:42:53.8384043Z Entering 'third_party/pthreadpool'
2025-07-17T08:42:53.8449259Z Entering 'third_party/pybind11'
2025-07-17T08:42:53.8513829Z Entering 'third_party/python-peachpy'
2025-07-17T08:42:53.8576400Z Entering 'third_party/sleef'
2025-07-17T08:42:53.8642973Z Entering 'third_party/tensorpipe'
2025-07-17T08:42:53.8712721Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-17T08:42:53.8768522Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-17T08:42:53.8832940Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-17T08:42:53.8888575Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-17T08:42:53.8951045Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-17T08:42:53.9042312Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2025-07-17T08:42:53.9412744Z Entering 'android/libs/fbjni'
2025-07-17T08:42:53.9478389Z Entering 'third_party/FP16'
2025-07-17T08:42:53.9546762Z Entering 'third_party/FXdiv'
2025-07-17T08:42:53.9614068Z Entering 'third_party/NNPACK'
2025-07-17T08:42:53.9679728Z Entering 'third_party/NVTX'
2025-07-17T08:42:53.9747277Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-17T08:42:53.9814342Z Entering 'third_party/XNNPACK'
2025-07-17T08:42:53.9898472Z Entering 'third_party/aiter'
2025-07-17T08:42:53.9959745Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-17T08:42:54.0039826Z Entering 'third_party/benchmark'
2025-07-17T08:42:54.0107089Z Entering 'third_party/composable_kernel'
2025-07-17T08:42:54.0177681Z Entering 'third_party/cpp-httplib'
2025-07-17T08:42:54.0242227Z Entering 'third_party/cpuinfo'
2025-07-17T08:42:54.0308955Z Entering 'third_party/cudnn_frontend'
2025-07-17T08:42:54.0375429Z Entering 'third_party/cutlass'
2025-07-17T08:42:54.0452451Z Entering 'third_party/fbgemm'
2025-07-17T08:42:54.0518015Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-17T08:42:54.0584898Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-17T08:42:54.0650460Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-17T08:42:54.0710241Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-17T08:42:54.0779237Z Entering 'third_party/fbgemm/external/googletest'
2025-07-17T08:42:54.0839445Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-17T08:42:54.0905446Z Entering 'third_party/fbgemm/external/json'
2025-07-17T08:42:54.0966101Z Entering 'third_party/flash-attention'
2025-07-17T08:42:54.1029648Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-17T08:42:54.1094608Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-17T08:42:54.1166525Z Entering 'third_party/flatbuffers'
2025-07-17T08:42:54.1232943Z Entering 'third_party/fmt'
2025-07-17T08:42:54.1298639Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-17T08:42:54.1363138Z Entering 'third_party/gloo'
2025-07-17T08:42:54.1430241Z Entering 'third_party/googletest'
2025-07-17T08:42:54.1497172Z Entering 'third_party/ideep'
2025-07-17T08:42:54.1560945Z Entering 'third_party/ideep/mkl-dnn'
2025-07-17T08:42:54.1640631Z Entering 'third_party/ittapi'
2025-07-17T08:42:54.1710063Z Entering 'third_party/kineto'
2025-07-17T08:42:54.1775522Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-17T08:42:54.1831425Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-17T08:42:54.1891404Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-17T08:42:54.1951079Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-17T08:42:54.2009357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-17T08:42:54.2067093Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-17T08:42:54.2126703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-17T08:42:54.2189702Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-17T08:42:54.2245856Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-17T08:42:54.2311234Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-17T08:42:54.2374544Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-17T08:42:54.2444677Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-17T08:42:54.2512892Z Entering 'third_party/kleidiai'
2025-07-17T08:42:54.2580424Z Entering 'third_party/mimalloc'
2025-07-17T08:42:54.2645972Z Entering 'third_party/nlohmann'
2025-07-17T08:42:54.2712205Z Entering 'third_party/onnx'
2025-07-17T08:42:54.2799096Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-17T08:42:54.2869683Z Entering 'third_party/opentelemetry-cpp'
2025-07-17T08:42:54.2930094Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-17T08:42:54.2991452Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-17T08:42:54.3046927Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-17T08:42:54.3110822Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-17T08:42:54.3168904Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-17T08:42:54.3231229Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-17T08:42:54.3286719Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-17T08:42:54.3349792Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-17T08:42:54.3409320Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-17T08:42:54.3476234Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-17T08:42:54.3560974Z Entering 'third_party/pocketfft'
2025-07-17T08:42:54.3627280Z Entering 'third_party/protobuf'
2025-07-17T08:42:54.3691862Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-17T08:42:54.3752783Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-17T08:42:54.3816130Z Entering 'third_party/psimd'
2025-07-17T08:42:54.3882792Z Entering 'third_party/pthreadpool'
2025-07-17T08:42:54.3951675Z Entering 'third_party/pybind11'
2025-07-17T08:42:54.4011660Z Entering 'third_party/python-peachpy'
2025-07-17T08:42:54.4075008Z Entering 'third_party/sleef'
2025-07-17T08:42:54.4142925Z Entering 'third_party/tensorpipe'
2025-07-17T08:42:54.4206599Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-17T08:42:54.4272167Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-17T08:42:54.4326870Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-17T08:42:54.4385821Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-17T08:42:54.4441025Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-17T08:42:54.4533677Z ##[endgroup]
2025-07-17T08:42:54.4642369Z [command]/usr/bin/git log -1 --format=%H
2025-07-17T08:42:54.4736810Z a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:42:54.4959032Z Prepare all required actions
2025-07-17T08:42:54.4959490Z Getting action download info
2025-07-17T08:42:54.6316259Z ##[group]Run ./.github/actions/setup-rocm
2025-07-17T08:42:54.6316548Z env:
2025-07-17T08:42:54.6316729Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:54.6316950Z ##[endgroup]
2025-07-17T08:42:54.6339091Z ##[group]Run dpkg -l | grep -E "  rocm"
2025-07-17T08:42:54.6339382Z dpkg -l | grep -E "  rocm"
2025-07-17T08:42:54.6365169Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:54.6365479Z env:
2025-07-17T08:42:54.6365658Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:54.6365866Z ##[endgroup]
2025-07-17T08:42:54.6596315Z ii  rocm                                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) software stack meta package
2025-07-17T08:42:54.6597251Z ii  rocm-cmake                             0.14.0.60303-74~22.04                   amd64        rocm-cmake built using CMake
2025-07-17T08:42:54.6598194Z ii  rocm-core                              6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6599124Z ii  rocm-dbgapi                            0.77.0.60303-74~22.04                   amd64        Library to provide AMD GPU debugger API
2025-07-17T08:42:54.6600118Z ii  rocm-debug-agent                       2.0.3.60303-74~22.04                    amd64        Radeon Open Compute Debug Agent (ROCdebug-agent)
2025-07-17T08:42:54.6601163Z ii  rocm-developer-tools                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6602165Z ii  rocm-device-libs                       1.0.0.60303-74~22.04                    amd64        Radeon Open Compute - device libraries
2025-07-17T08:42:54.6602996Z ii  rocm-gdb                               15.2.60303-74~22.04                     amd64        ROCgdb
2025-07-17T08:42:54.6603966Z ii  rocm-hip-libraries                     6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6604808Z ii  rocm-hip-runtime                       6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6605427Z ii  rocm-hip-runtime-dev                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6606012Z ii  rocm-hip-sdk                           6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6606619Z ii  rocm-language-runtime                  6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6607195Z ii  rocm-llvm                              18.0.0.25012.60303-74~22.04             amd64        ROCm core compiler
2025-07-17T08:42:54.6607738Z ii  rocm-ml-libraries                      6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6608320Z ii  rocm-ml-sdk                            6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6608833Z ii  rocm-opencl                            2.0.0.60303-74~22.04                    amd64        clr built using CMake
2025-07-17T08:42:54.6609311Z ii  rocm-opencl-dev                        2.0.0.60303-74~22.04                    amd64        clr built using CMake
2025-07-17T08:42:54.6609871Z ii  rocm-opencl-runtime                    6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6610465Z ii  rocm-opencl-sdk                        6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6611645Z ii  rocm-openmp-sdk                        6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) OpenMP Software development Kit.
2025-07-17T08:42:54.6612263Z ii  rocm-smi-lib                           7.4.0.60303-74~22.04                    amd64        AMD System Management libraries
2025-07-17T08:42:54.6612784Z ii  rocm-utils                             6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-17T08:42:54.6613332Z ii  rocminfo                               1.0.0.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime rocminfo tool
2025-07-17T08:42:54.6634029Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-07-17T08:42:54.6634517Z # ignore expansion of "docker ps -q" since it could be empty
2025-07-17T08:42:54.6634881Z # shellcheck disable=SC2046
2025-07-17T08:42:54.6635175Z docker stop $(docker ps -q) || true
2025-07-17T08:42:54.6635459Z # Prune all stopped containers.
2025-07-17T08:42:54.6635744Z docker container prune -f
2025-07-17T08:42:54.6663246Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:54.6663571Z env:
2025-07-17T08:42:54.6663760Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:54.6663973Z ##[endgroup]
2025-07-17T08:42:54.7091923Z "docker stop" requires at least 1 argument.
2025-07-17T08:42:54.7092511Z See 'docker stop --help'.
2025-07-17T08:42:54.7092762Z 
2025-07-17T08:42:54.7093001Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2025-07-17T08:42:54.7093392Z 
2025-07-17T08:42:54.7093555Z Stop one or more running containers
2025-07-17T08:42:54.7300476Z Total reclaimed space: 0B
2025-07-17T08:42:54.7358500Z ##[group]Run cat /etc/os-release || true
2025-07-17T08:42:54.7358825Z cat /etc/os-release || true
2025-07-17T08:42:54.7359154Z cat /etc/apt/sources.list.d/rocm.list || true
2025-07-17T08:42:54.7359483Z cat /opt/rocm/.info/version || true
2025-07-17T08:42:54.7359742Z whoami
2025-07-17T08:42:54.7386471Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:54.7386796Z env:
2025-07-17T08:42:54.7386982Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:54.7387203Z ##[endgroup]
2025-07-17T08:42:54.7449569Z PRETTY_NAME="Ubuntu 22.04.4 LTS"
2025-07-17T08:42:54.7450029Z NAME="Ubuntu"
2025-07-17T08:42:54.7450340Z VERSION_ID="22.04"
2025-07-17T08:42:54.7450708Z VERSION="22.04.4 LTS (Jammy Jellyfish)"
2025-07-17T08:42:54.7451137Z VERSION_CODENAME=jammy
2025-07-17T08:42:54.7451470Z ID=ubuntu
2025-07-17T08:42:54.7451738Z ID_LIKE=debian
2025-07-17T08:42:54.7452089Z HOME_URL="https://www.ubuntu.com/"
2025-07-17T08:42:54.7452578Z SUPPORT_URL="https://help.ubuntu.com/"
2025-07-17T08:42:54.7453113Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2025-07-17T08:42:54.7453915Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2025-07-17T08:42:54.7454635Z UBUNTU_CODENAME=jammy
2025-07-17T08:42:54.7464818Z deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3.3 jammy main
2025-07-17T08:42:54.7479257Z 6.3.3-74
2025-07-17T08:42:54.7505882Z pytorchci
2025-07-17T08:42:54.7530329Z ##[group]Run dpkg -l | grep -E "  amdgpu"
2025-07-17T08:42:54.7530641Z dpkg -l | grep -E "  amdgpu"
2025-07-17T08:42:54.7553976Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:54.7554313Z env:
2025-07-17T08:42:54.7554508Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:54.7554724Z ##[endgroup]
2025-07-17T08:42:54.7780671Z ii  amdgpu-core                            1:6.3.60303-2119913.22.04               all          Core meta package for unified amdgpu driver.
2025-07-17T08:42:54.7781651Z ii  amdgpu-dkms                            1:6.10.5.60303-2119913.22.04            all          amdgpu driver in DKMS format.
2025-07-17T08:42:54.7783141Z ii  amdgpu-dkms-firmware                   1:6.10.5.60303-2119913.22.04            all          firmware blobs used by amdgpu driver in DKMS format
2025-07-17T08:42:54.7784553Z ii  amdgpu-install                         6.3.60303-2119913.22.04                 all          AMDGPU driver repository and installer
2025-07-17T08:42:54.7811678Z ##[group]Run rocm-smi
2025-07-17T08:42:54.7811943Z rocm-smi
2025-07-17T08:42:54.7838235Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:54.7838571Z env:
2025-07-17T08:42:54.7838765Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:54.7838981Z ##[endgroup]
2025-07-17T08:42:54.9387979Z 
2025-07-17T08:42:54.9388142Z 
2025-07-17T08:42:54.9388605Z ========================================= ROCm System Management Interface =========================================
2025-07-17T08:42:54.9389386Z =================================================== Concise Info ===================================================
2025-07-17T08:42:54.9390274Z Device  Node  IDs              Temp    Power  Partitions          SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
2025-07-17T08:42:54.9391623Z               (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)                                                   
2025-07-17T08:42:54.9392310Z ====================================================================================================================
2025-07-17T08:42:54.9393316Z 0       4     0x740c,   57586  35.0°C  98.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-07-17T08:42:54.9394279Z 1       5     0x740c,   45873  35.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-07-17T08:42:54.9395230Z 2       2     0x740c,   51627  44.0°C  99.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-07-17T08:42:54.9396161Z 3       3     0x740c,   64489  24.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-07-17T08:42:54.9397082Z 4       8     0x740c,   30939  34.0°C  92.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-07-17T08:42:54.9398015Z 5       9     0x740c,   8466   35.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-07-17T08:42:54.9398885Z 6       6     0x740c,   41154  31.0°C  95.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-07-17T08:42:54.9399817Z 7       7     0x740c,   63755  27.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-07-17T08:42:54.9404179Z ====================================================================================================================
2025-07-17T08:42:54.9404528Z =============================================== End of ROCm SMI Log ================================================
2025-07-17T08:42:54.9554852Z ##[group]Run rocminfo
2025-07-17T08:42:54.9555103Z rocminfo
2025-07-17T08:42:54.9581608Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:54.9581961Z env:
2025-07-17T08:42:54.9582147Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:54.9582359Z ##[endgroup]
2025-07-17T08:42:55.0889871Z ROCk module version 6.10.5 is loaded
2025-07-17T08:42:55.0890291Z =====================    
2025-07-17T08:42:55.0890593Z HSA System Attributes    
2025-07-17T08:42:55.0890886Z =====================    
2025-07-17T08:42:55.0891164Z Runtime Version:         1.14
2025-07-17T08:42:55.0891493Z Runtime Ext Version:     1.6
2025-07-17T08:42:55.0891840Z System Timestamp Freq.:  1000.000000MHz
2025-07-17T08:42:55.0892612Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-07-17T08:42:55.0893614Z Machine Model:           LARGE                              
2025-07-17T08:42:55.0894226Z System Endianness:       LITTLE                             
2025-07-17T08:42:55.0894752Z Mwaitx:                  DISABLED
2025-07-17T08:42:55.0895383Z DMAbuf Support:          YES
2025-07-17T08:42:55.0895633Z 
2025-07-17T08:42:55.0896341Z ==========               
2025-07-17T08:42:55.0896709Z HSA Agents               
2025-07-17T08:42:55.0897061Z ==========               
2025-07-17T08:42:55.0897649Z *******                  
2025-07-17T08:42:55.0897990Z Agent 1                  
2025-07-17T08:42:55.0898318Z *******                  
2025-07-17T08:42:55.0898755Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-07-17T08:42:55.0899319Z   Uuid:                    CPU-XX                             
2025-07-17T08:42:55.0899857Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-17T08:42:55.0900417Z   Vendor Name:             CPU                                
2025-07-17T08:42:55.0900945Z   Feature:                 None specified                     
2025-07-17T08:42:55.0901478Z   Profile:                 FULL_PROFILE                       
2025-07-17T08:42:55.0902080Z   Float Round Mode:        NEAR                               
2025-07-17T08:42:55.0902814Z   Max Queue Number:        0(0x0)                             
2025-07-17T08:42:55.0903345Z   Queue Min Size:          0(0x0)                             
2025-07-17T08:42:55.0903757Z   Queue Max Size:          0(0x0)                             
2025-07-17T08:42:55.0904169Z   Queue Type:              MULTI                              
2025-07-17T08:42:55.0904544Z   Node:                    0                                  
2025-07-17T08:42:55.0904931Z   Device Type:             CPU                                
2025-07-17T08:42:55.0905299Z   Cache Info:              
2025-07-17T08:42:55.0905589Z     L1:                      32768(0x8000) KB                   
2025-07-17T08:42:55.0905962Z   Chip ID:                 0(0x0)                             
2025-07-17T08:42:55.0906348Z   ASIC Revision:           0(0x0)                             
2025-07-17T08:42:55.0906774Z   Cacheline Size:          64(0x40)                           
2025-07-17T08:42:55.0907189Z   Max Clock Freq. (MHz):   2000                               
2025-07-17T08:42:55.0907589Z   BDFID:                   0                                  
2025-07-17T08:42:55.0907978Z   Internal Node ID:        0                                  
2025-07-17T08:42:55.0908397Z   Compute Unit:            64                                 
2025-07-17T08:42:55.0908801Z   SIMDs per CU:            0                                  
2025-07-17T08:42:55.0909204Z   Shader Engines:          0                                  
2025-07-17T08:42:55.0909635Z   Shader Arrs. per Eng.:   0                                  
2025-07-17T08:42:55.0910068Z   WatchPts on Addr. Ranges:1                                  
2025-07-17T08:42:55.0910458Z   Memory Properties:       
2025-07-17T08:42:55.0910730Z   Features:                None
2025-07-17T08:42:55.0911007Z   Pool Info:               
2025-07-17T08:42:55.0911271Z     Pool 1                   
2025-07-17T08:42:55.0911607Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:42:55.0912021Z       Size:                    528249800(0x1f7c73c8) KB           
2025-07-17T08:42:55.0912440Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0912875Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0913321Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:42:55.0913692Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0914019Z       Accessible by all:       TRUE                               
2025-07-17T08:42:55.0914306Z     Pool 2                   
2025-07-17T08:42:55.0914568Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:42:55.0914876Z       Size:                    528249800(0x1f7c73c8) KB           
2025-07-17T08:42:55.0915178Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0915497Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0915840Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:42:55.0916348Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0916803Z       Accessible by all:       TRUE                               
2025-07-17T08:42:55.0917098Z     Pool 3                   
2025-07-17T08:42:55.0917354Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-07-17T08:42:55.0917663Z       Size:                    528249800(0x1f7c73c8) KB           
2025-07-17T08:42:55.0917962Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0918287Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0918614Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:42:55.0918959Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0919294Z       Accessible by all:       TRUE                               
2025-07-17T08:42:55.0919570Z     Pool 4                   
2025-07-17T08:42:55.0919837Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:42:55.0920139Z       Size:                    528249800(0x1f7c73c8) KB           
2025-07-17T08:42:55.0920453Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0920771Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0921114Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:42:55.0921454Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0921779Z       Accessible by all:       TRUE                               
2025-07-17T08:42:55.0922071Z   ISA Info:                
2025-07-17T08:42:55.0922270Z *******                  
2025-07-17T08:42:55.0922476Z Agent 2                  
2025-07-17T08:42:55.0922662Z *******                  
2025-07-17T08:42:55.0922905Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-07-17T08:42:55.0923207Z   Uuid:                    CPU-XX                             
2025-07-17T08:42:55.0923535Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-17T08:42:55.0923872Z   Vendor Name:             CPU                                
2025-07-17T08:42:55.0924186Z   Feature:                 None specified                     
2025-07-17T08:42:55.0924505Z   Profile:                 FULL_PROFILE                       
2025-07-17T08:42:55.0924858Z   Float Round Mode:        NEAR                               
2025-07-17T08:42:55.0925186Z   Max Queue Number:        0(0x0)                             
2025-07-17T08:42:55.0925500Z   Queue Min Size:          0(0x0)                             
2025-07-17T08:42:55.0925815Z   Queue Max Size:          0(0x0)                             
2025-07-17T08:42:55.0926123Z   Queue Type:              MULTI                              
2025-07-17T08:42:55.0926405Z   Node:                    1                                  
2025-07-17T08:42:55.0926700Z   Device Type:             CPU                                
2025-07-17T08:42:55.0926980Z   Cache Info:              
2025-07-17T08:42:55.0927209Z     L1:                      32768(0x8000) KB                   
2025-07-17T08:42:55.0927494Z   Chip ID:                 0(0x0)                             
2025-07-17T08:42:55.0927801Z   ASIC Revision:           0(0x0)                             
2025-07-17T08:42:55.0928122Z   Cacheline Size:          64(0x40)                           
2025-07-17T08:42:55.0928441Z   Max Clock Freq. (MHz):   2000                               
2025-07-17T08:42:55.0928748Z   BDFID:                   0                                  
2025-07-17T08:42:55.0929048Z   Internal Node ID:        1                                  
2025-07-17T08:42:55.0929363Z   Compute Unit:            64                                 
2025-07-17T08:42:55.0929662Z   SIMDs per CU:            0                                  
2025-07-17T08:42:55.0929977Z   Shader Engines:          0                                  
2025-07-17T08:42:55.0930445Z   Shader Arrs. per Eng.:   0                                  
2025-07-17T08:42:55.0930790Z   WatchPts on Addr. Ranges:1                                  
2025-07-17T08:42:55.0931211Z   Memory Properties:       
2025-07-17T08:42:55.0931433Z   Features:                None
2025-07-17T08:42:55.0931646Z   Pool Info:               
2025-07-17T08:42:55.0931844Z     Pool 1                   
2025-07-17T08:42:55.0932107Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:42:55.0932419Z       Size:                    528402424(0x1f7ec7f8) KB           
2025-07-17T08:42:55.0932734Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0933063Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0933400Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:42:55.0933750Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0934076Z       Accessible by all:       TRUE                               
2025-07-17T08:42:55.0934374Z     Pool 2                   
2025-07-17T08:42:55.0934634Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:42:55.0934956Z       Size:                    528402424(0x1f7ec7f8) KB           
2025-07-17T08:42:55.0935258Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0935586Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0935997Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:42:55.0936339Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0936674Z       Accessible by all:       TRUE                               
2025-07-17T08:42:55.0936950Z     Pool 3                   
2025-07-17T08:42:55.0937243Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-07-17T08:42:55.0937563Z       Size:                    528402424(0x1f7ec7f8) KB           
2025-07-17T08:42:55.0937906Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0938260Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0938596Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:42:55.0938975Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0939323Z       Accessible by all:       TRUE                               
2025-07-17T08:42:55.0939626Z     Pool 4                   
2025-07-17T08:42:55.0939891Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:42:55.0940208Z       Size:                    528402424(0x1f7ec7f8) KB           
2025-07-17T08:42:55.0940544Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0940878Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0941240Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:42:55.0941591Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0941932Z       Accessible by all:       TRUE                               
2025-07-17T08:42:55.0942221Z   ISA Info:                
2025-07-17T08:42:55.0942416Z *******                  
2025-07-17T08:42:55.0942622Z Agent 3                  
2025-07-17T08:42:55.0942807Z *******                  
2025-07-17T08:42:55.0943027Z   Name:                    gfx90a                             
2025-07-17T08:42:55.0943314Z   Uuid:                    GPU-1635b42468671753               
2025-07-17T08:42:55.0943643Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:42:55.0943963Z   Vendor Name:             AMD                                
2025-07-17T08:42:55.0944285Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:42:55.0944599Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:42:55.0944914Z   Float Round Mode:        NEAR                               
2025-07-17T08:42:55.0945407Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:42:55.0945883Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:42:55.0946205Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:42:55.0946514Z   Queue Type:              MULTI                              
2025-07-17T08:42:55.0946812Z   Node:                    2                                  
2025-07-17T08:42:55.0947112Z   Device Type:             GPU                                
2025-07-17T08:42:55.0947381Z   Cache Info:              
2025-07-17T08:42:55.0947610Z     L1:                      16(0x10) KB                        
2025-07-17T08:42:55.0947883Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:42:55.0948174Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:42:55.0948477Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:42:55.0948812Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:42:55.0949137Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:42:55.0949452Z   BDFID:                   12800                              
2025-07-17T08:42:55.0949762Z   Internal Node ID:        2                                  
2025-07-17T08:42:55.0950176Z   Compute Unit:            104                                
2025-07-17T08:42:55.0950762Z   SIMDs per CU:            4                                  
2025-07-17T08:42:55.0951207Z   Shader Engines:          8                                  
2025-07-17T08:42:55.0951657Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:42:55.0952055Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:42:55.0952654Z   Coherent Host Access:    FALSE                              
2025-07-17T08:42:55.0953077Z   Memory Properties:       
2025-07-17T08:42:55.0953362Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:42:55.0953889Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:42:55.0954301Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:42:55.0954728Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.0955214Z   Workgroup Max Size per Dimension:
2025-07-17T08:42:55.0955594Z     x                        1024(0x400)                        
2025-07-17T08:42:55.0955962Z     y                        1024(0x400)                        
2025-07-17T08:42:55.0956414Z     z                        1024(0x400)                        
2025-07-17T08:42:55.0956831Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:42:55.0957234Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:42:55.0957774Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.0958147Z   Grid Max Size per Dimension:
2025-07-17T08:42:55.0958512Z     x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.0958996Z     y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.0959353Z     z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.0959806Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:42:55.0965526Z   Packet Processor uCode:: 92                                 
2025-07-17T08:42:55.0966102Z   SDMA engine uCode::      9                                  
2025-07-17T08:42:55.0966623Z   IOMMU Support::          None                               
2025-07-17T08:42:55.0967063Z   Pool Info:               
2025-07-17T08:42:55.0967468Z     Pool 1                   
2025-07-17T08:42:55.0967858Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:42:55.0968375Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.0968826Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0969539Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0970170Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.0970652Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0971118Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.0971465Z     Pool 2                   
2025-07-17T08:42:55.0971987Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:42:55.0972397Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.0972827Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0973390Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0973812Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.0974258Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0974785Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.0975196Z     Pool 3                   
2025-07-17T08:42:55.0975539Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:42:55.0976147Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.0976586Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.0976983Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.0977544Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.0977963Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.0978411Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.0978904Z     Pool 4                   
2025-07-17T08:42:55.0979237Z       Segment:                 GROUP                              
2025-07-17T08:42:55.0979679Z       Size:                    64(0x40) KB                        
2025-07-17T08:42:55.0980152Z       Allocatable:             FALSE                              
2025-07-17T08:42:55.0980635Z       Alloc Granule:           0KB                                
2025-07-17T08:42:55.0981049Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:42:55.0981579Z       Alloc Alignment:         0KB                                
2025-07-17T08:42:55.0982064Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.0982437Z   ISA Info:                
2025-07-17T08:42:55.0982859Z     ISA 1                    
2025-07-17T08:42:55.0983219Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:42:55.0983723Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:42:55.0984114Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:42:55.0984653Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.0985155Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.0985533Z       Fast f16:                TRUE                               
2025-07-17T08:42:55.0986052Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.0986451Z       Workgroup Max Size per Dimension:
2025-07-17T08:42:55.0986813Z         x                        1024(0x400)                        
2025-07-17T08:42:55.0987269Z         y                        1024(0x400)                        
2025-07-17T08:42:55.0987622Z         z                        1024(0x400)                        
2025-07-17T08:42:55.0988006Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.0988445Z       Grid Max Size per Dimension:
2025-07-17T08:42:55.0988819Z         x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.0989155Z         y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.0989783Z         z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.0990364Z       FBarrier Max Size:       32                                 
2025-07-17T08:42:55.0990729Z *******                  
2025-07-17T08:42:55.0991130Z Agent 4                  
2025-07-17T08:42:55.0991392Z *******                  
2025-07-17T08:42:55.0991715Z   Name:                    gfx90a                             
2025-07-17T08:42:55.0992145Z   Uuid:                    GPU-8cfe31f8885bf623               
2025-07-17T08:42:55.0992598Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:42:55.0993031Z   Vendor Name:             AMD                                
2025-07-17T08:42:55.0993471Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:42:55.0993917Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:42:55.0994302Z   Float Round Mode:        NEAR                               
2025-07-17T08:42:55.0994813Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:42:55.0995218Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:42:55.0995641Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:42:55.0996113Z   Queue Type:              MULTI                              
2025-07-17T08:42:55.0996503Z   Node:                    3                                  
2025-07-17T08:42:55.0996912Z   Device Type:             GPU                                
2025-07-17T08:42:55.0997260Z   Cache Info:              
2025-07-17T08:42:55.0997675Z     L1:                      16(0x10) KB                        
2025-07-17T08:42:55.0998013Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:42:55.0998411Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:42:55.0998917Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:42:55.0999309Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:42:55.0999764Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:42:55.1000230Z   BDFID:                   13568                              
2025-07-17T08:42:55.1000668Z   Internal Node ID:        3                                  
2025-07-17T08:42:55.1001061Z   Compute Unit:            104                                
2025-07-17T08:42:55.1001537Z   SIMDs per CU:            4                                  
2025-07-17T08:42:55.1001992Z   Shader Engines:          8                                  
2025-07-17T08:42:55.1002380Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:42:55.1002878Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:42:55.1003305Z   Coherent Host Access:    FALSE                              
2025-07-17T08:42:55.1003718Z   Memory Properties:       
2025-07-17T08:42:55.1004121Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:42:55.1004513Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:42:55.1004957Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:42:55.1005396Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1005842Z   Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1006162Z     x                        1024(0x400)                        
2025-07-17T08:42:55.1006576Z     y                        1024(0x400)                        
2025-07-17T08:42:55.1006990Z     z                        1024(0x400)                        
2025-07-17T08:42:55.1007351Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:42:55.1007820Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:42:55.1008262Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1008655Z   Grid Max Size per Dimension:
2025-07-17T08:42:55.1008932Z     x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1009561Z     y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1009944Z     z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1010440Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:42:55.1011004Z   Packet Processor uCode:: 92                                 
2025-07-17T08:42:55.1011412Z   SDMA engine uCode::      9                                  
2025-07-17T08:42:55.1011855Z   IOMMU Support::          None                               
2025-07-17T08:42:55.1012341Z   Pool Info:               
2025-07-17T08:42:55.1012622Z     Pool 1                   
2025-07-17T08:42:55.1012998Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:42:55.1013481Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1013906Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1014319Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1014851Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1015345Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1015751Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1016285Z     Pool 2                   
2025-07-17T08:42:55.1016616Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:42:55.1017053Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1017482Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1017964Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1018410Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1018863Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1019342Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1019698Z     Pool 3                   
2025-07-17T08:42:55.1020158Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:42:55.1020580Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1020994Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1021473Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1021913Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1022363Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1022744Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1023247Z     Pool 4                   
2025-07-17T08:42:55.1023558Z       Segment:                 GROUP                              
2025-07-17T08:42:55.1023940Z       Size:                    64(0x40) KB                        
2025-07-17T08:42:55.1024465Z       Allocatable:             FALSE                              
2025-07-17T08:42:55.1024864Z       Alloc Granule:           0KB                                
2025-07-17T08:42:55.1025283Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:42:55.1049139Z       Alloc Alignment:         0KB                                
2025-07-17T08:42:55.1049536Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1049853Z   ISA Info:                
2025-07-17T08:42:55.1050062Z     ISA 1                    
2025-07-17T08:42:55.1050337Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:42:55.1050717Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:42:55.1051055Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:42:55.1051399Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1051728Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1052339Z       Fast f16:                TRUE                               
2025-07-17T08:42:55.1052803Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1053127Z       Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1053396Z         x                        1024(0x400)                        
2025-07-17T08:42:55.1053662Z         y                        1024(0x400)                        
2025-07-17T08:42:55.1053919Z         z                        1024(0x400)                        
2025-07-17T08:42:55.1054202Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1054492Z       Grid Max Size per Dimension:
2025-07-17T08:42:55.1054729Z         x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1055004Z         y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1055280Z         z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1055581Z       FBarrier Max Size:       32                                 
2025-07-17T08:42:55.1055864Z *******                  
2025-07-17T08:42:55.1056145Z Agent 5                  
2025-07-17T08:42:55.1056338Z *******                  
2025-07-17T08:42:55.1056552Z   Name:                    gfx90a                             
2025-07-17T08:42:55.1056854Z   Uuid:                    GPU-1aa655a92f77fcda               
2025-07-17T08:42:55.1057170Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:42:55.1057500Z   Vendor Name:             AMD                                
2025-07-17T08:42:55.1057812Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:42:55.1058112Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:42:55.1058426Z   Float Round Mode:        NEAR                               
2025-07-17T08:42:55.1058741Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:42:55.1059065Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:42:55.1059363Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:42:55.1059676Z   Queue Type:              MULTI                              
2025-07-17T08:42:55.1059972Z   Node:                    4                                  
2025-07-17T08:42:55.1060262Z   Device Type:             GPU                                
2025-07-17T08:42:55.1060540Z   Cache Info:              
2025-07-17T08:42:55.1060769Z     L1:                      16(0x10) KB                        
2025-07-17T08:42:55.1061043Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:42:55.1061310Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:42:55.1061612Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:42:55.1061931Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:42:55.1062243Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:42:55.1062541Z   BDFID:                   4352                               
2025-07-17T08:42:55.1062850Z   Internal Node ID:        4                                  
2025-07-17T08:42:55.1063162Z   Compute Unit:            104                                
2025-07-17T08:42:55.1063457Z   SIMDs per CU:            4                                  
2025-07-17T08:42:55.1063764Z   Shader Engines:          8                                  
2025-07-17T08:42:55.1064081Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:42:55.1064408Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:42:55.1064735Z   Coherent Host Access:    FALSE                              
2025-07-17T08:42:55.1065022Z   Memory Properties:       
2025-07-17T08:42:55.1065249Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:42:55.1065544Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:42:55.1066103Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:42:55.1066439Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1066879Z   Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1067135Z     x                        1024(0x400)                        
2025-07-17T08:42:55.1067401Z     y                        1024(0x400)                        
2025-07-17T08:42:55.1067658Z     z                        1024(0x400)                        
2025-07-17T08:42:55.1067939Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:42:55.1068254Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:42:55.1068577Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1068870Z   Grid Max Size per Dimension:
2025-07-17T08:42:55.1069100Z     x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1069367Z     y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1069629Z     z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1069948Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:42:55.1070312Z   Packet Processor uCode:: 92                                 
2025-07-17T08:42:55.1070651Z   SDMA engine uCode::      9                                  
2025-07-17T08:42:55.1070988Z   IOMMU Support::          None                               
2025-07-17T08:42:55.1071259Z   Pool Info:               
2025-07-17T08:42:55.1071473Z     Pool 1                   
2025-07-17T08:42:55.1071747Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:42:55.1072072Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1072373Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1072720Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1073308Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1073666Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1074009Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1074307Z     Pool 2                   
2025-07-17T08:42:55.1074563Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:42:55.1074877Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1075174Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1075490Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1075827Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1076158Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1076487Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1076766Z     Pool 3                   
2025-07-17T08:42:55.1077020Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:42:55.1077317Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1077615Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1077928Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1078257Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1078591Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1078909Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1079191Z     Pool 4                   
2025-07-17T08:42:55.1079423Z       Segment:                 GROUP                              
2025-07-17T08:42:55.1079713Z       Size:                    64(0x40) KB                        
2025-07-17T08:42:55.1080002Z       Allocatable:             FALSE                              
2025-07-17T08:42:55.1080473Z       Alloc Granule:           0KB                                
2025-07-17T08:42:55.1080925Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:42:55.1081268Z       Alloc Alignment:         0KB                                
2025-07-17T08:42:55.1081602Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1081878Z   ISA Info:                
2025-07-17T08:42:55.1082084Z     ISA 1                    
2025-07-17T08:42:55.1082344Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:42:55.1082701Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:42:55.1083041Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:42:55.1083365Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1083709Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1084016Z       Fast f16:                TRUE                               
2025-07-17T08:42:55.1084338Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1084639Z       Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1084907Z         x                        1024(0x400)                        
2025-07-17T08:42:55.1085172Z         y                        1024(0x400)                        
2025-07-17T08:42:55.1085425Z         z                        1024(0x400)                        
2025-07-17T08:42:55.1085710Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1085992Z       Grid Max Size per Dimension:
2025-07-17T08:42:55.1086231Z         x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1086489Z         y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1086754Z         z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1087051Z       FBarrier Max Size:       32                                 
2025-07-17T08:42:55.1087329Z *******                  
2025-07-17T08:42:55.1087523Z Agent 6                  
2025-07-17T08:42:55.1087709Z *******                  
2025-07-17T08:42:55.1087927Z   Name:                    gfx90a                             
2025-07-17T08:42:55.1088225Z   Uuid:                    GPU-aaccf1cf8b4ae211               
2025-07-17T08:42:55.1088551Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:42:55.1088877Z   Vendor Name:             AMD                                
2025-07-17T08:42:55.1089185Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:42:55.1089494Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:42:55.1089804Z   Float Round Mode:        NEAR                               
2025-07-17T08:42:55.1090126Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:42:55.1090432Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:42:55.1090744Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:42:55.1091046Z   Queue Type:              MULTI                              
2025-07-17T08:42:55.1091336Z   Node:                    5                                  
2025-07-17T08:42:55.1091627Z   Device Type:             GPU                                
2025-07-17T08:42:55.1091896Z   Cache Info:              
2025-07-17T08:42:55.1092126Z     L1:                      16(0x10) KB                        
2025-07-17T08:42:55.1092386Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:42:55.1092671Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:42:55.1092964Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:42:55.1093282Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:42:55.1093595Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:42:55.1093891Z   BDFID:                   5120                               
2025-07-17T08:42:55.1094329Z   Internal Node ID:        5                                  
2025-07-17T08:42:55.1094777Z   Compute Unit:            104                                
2025-07-17T08:42:55.1095092Z   SIMDs per CU:            4                                  
2025-07-17T08:42:55.1095392Z   Shader Engines:          8                                  
2025-07-17T08:42:55.1095717Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:42:55.1096109Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:42:55.1096445Z   Coherent Host Access:    FALSE                              
2025-07-17T08:42:55.1096749Z   Memory Properties:       
2025-07-17T08:42:55.1096979Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:42:55.1097276Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:42:55.1097589Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:42:55.1097928Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1098219Z   Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1098475Z     x                        1024(0x400)                        
2025-07-17T08:42:55.1098739Z     y                        1024(0x400)                        
2025-07-17T08:42:55.1098988Z     z                        1024(0x400)                        
2025-07-17T08:42:55.1099275Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:42:55.1099588Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:42:55.1099907Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1100183Z   Grid Max Size per Dimension:
2025-07-17T08:42:55.1100411Z     x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1100669Z     y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1100925Z     z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1101235Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:42:55.1101577Z   Packet Processor uCode:: 92                                 
2025-07-17T08:42:55.1101913Z   SDMA engine uCode::      9                                  
2025-07-17T08:42:55.1102229Z   IOMMU Support::          None                               
2025-07-17T08:42:55.1102510Z   Pool Info:               
2025-07-17T08:42:55.1102707Z     Pool 1                   
2025-07-17T08:42:55.1102975Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:42:55.1103289Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1103592Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1103918Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1104253Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1104594Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1104927Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1105211Z     Pool 2                   
2025-07-17T08:42:55.1105468Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:42:55.1105767Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1106071Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1106386Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1106720Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1107057Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1107390Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1107673Z     Pool 3                   
2025-07-17T08:42:55.1107910Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:42:55.1108372Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1108817Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1109145Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1109496Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1109839Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1110185Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1110472Z     Pool 4                   
2025-07-17T08:42:55.1110710Z       Segment:                 GROUP                              
2025-07-17T08:42:55.1111011Z       Size:                    64(0x40) KB                        
2025-07-17T08:42:55.1111309Z       Allocatable:             FALSE                              
2025-07-17T08:42:55.1111649Z       Alloc Granule:           0KB                                
2025-07-17T08:42:55.1111998Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:42:55.1112356Z       Alloc Alignment:         0KB                                
2025-07-17T08:42:55.1112695Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1112971Z   ISA Info:                
2025-07-17T08:42:55.1113177Z     ISA 1                    
2025-07-17T08:42:55.1113438Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:42:55.1113788Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:42:55.1114117Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:42:55.1114460Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1114806Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1115122Z       Fast f16:                TRUE                               
2025-07-17T08:42:55.1115445Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1115742Z       Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1116011Z         x                        1024(0x400)                        
2025-07-17T08:42:55.1116268Z         y                        1024(0x400)                        
2025-07-17T08:42:55.1116530Z         z                        1024(0x400)                        
2025-07-17T08:42:55.1116822Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1117104Z       Grid Max Size per Dimension:
2025-07-17T08:42:55.1117348Z         x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1117609Z         y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1117872Z         z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1118162Z       FBarrier Max Size:       32                                 
2025-07-17T08:42:55.1118443Z *******                  
2025-07-17T08:42:55.1118638Z Agent 7                  
2025-07-17T08:42:55.1118832Z *******                  
2025-07-17T08:42:55.1119063Z   Name:                    gfx90a                             
2025-07-17T08:42:55.1119355Z   Uuid:                    GPU-27c44677f57701f4               
2025-07-17T08:42:55.1119681Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:42:55.1120004Z   Vendor Name:             AMD                                
2025-07-17T08:42:55.1120321Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:42:55.1120628Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:42:55.1120946Z   Float Round Mode:        NEAR                               
2025-07-17T08:42:55.1121272Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:42:55.1121577Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:42:55.1121883Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:42:55.1122317Z   Queue Type:              MULTI                              
2025-07-17T08:42:55.1122726Z   Node:                    6                                  
2025-07-17T08:42:55.1123023Z   Device Type:             GPU                                
2025-07-17T08:42:55.1123303Z   Cache Info:              
2025-07-17T08:42:55.1123520Z     L1:                      16(0x10) KB                        
2025-07-17T08:42:55.1123793Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:42:55.1124081Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:42:55.1124374Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:42:55.1124703Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:42:55.1125014Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:42:55.1125317Z   BDFID:                   44544                              
2025-07-17T08:42:55.1125626Z   Internal Node ID:        6                                  
2025-07-17T08:42:55.1125950Z   Compute Unit:            104                                
2025-07-17T08:42:55.1126272Z   SIMDs per CU:            4                                  
2025-07-17T08:42:55.1126583Z   Shader Engines:          8                                  
2025-07-17T08:42:55.1126919Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:42:55.1127251Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:42:55.1127599Z   Coherent Host Access:    FALSE                              
2025-07-17T08:42:55.1127895Z   Memory Properties:       
2025-07-17T08:42:55.1128135Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:42:55.1128444Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:42:55.1128770Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:42:55.1129113Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1129427Z   Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1129687Z     x                        1024(0x400)                        
2025-07-17T08:42:55.1129956Z     y                        1024(0x400)                        
2025-07-17T08:42:55.1130220Z     z                        1024(0x400)                        
2025-07-17T08:42:55.1130497Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:42:55.1130816Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:42:55.1131154Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1131442Z   Grid Max Size per Dimension:
2025-07-17T08:42:55.1131684Z     x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1131951Z     y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1132226Z     z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1132524Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:42:55.1132889Z   Packet Processor uCode:: 92                                 
2025-07-17T08:42:55.1133236Z   SDMA engine uCode::      9                                  
2025-07-17T08:42:55.1133558Z   IOMMU Support::          None                               
2025-07-17T08:42:55.1133842Z   Pool Info:               
2025-07-17T08:42:55.1134046Z     Pool 1                   
2025-07-17T08:42:55.1134314Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:42:55.1134633Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1134952Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1135278Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1135611Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1136039Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1136534Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1136831Z     Pool 2                   
2025-07-17T08:42:55.1137232Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:42:55.1137563Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1137880Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1138204Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1138555Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1138897Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1139235Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1139517Z     Pool 3                   
2025-07-17T08:42:55.1139773Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:42:55.1140077Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1140385Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1140719Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1141051Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1141396Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1141732Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1142016Z     Pool 4                   
2025-07-17T08:42:55.1142254Z       Segment:                 GROUP                              
2025-07-17T08:42:55.1142538Z       Size:                    64(0x40) KB                        
2025-07-17T08:42:55.1142841Z       Allocatable:             FALSE                              
2025-07-17T08:42:55.1143169Z       Alloc Granule:           0KB                                
2025-07-17T08:42:55.1143502Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:42:55.1143835Z       Alloc Alignment:         0KB                                
2025-07-17T08:42:55.1144172Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1144447Z   ISA Info:                
2025-07-17T08:42:55.1144648Z     ISA 1                    
2025-07-17T08:42:55.1144907Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:42:55.1145251Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:42:55.1145583Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:42:55.1145908Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1146245Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1146552Z       Fast f16:                TRUE                               
2025-07-17T08:42:55.1146863Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1147166Z       Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1147418Z         x                        1024(0x400)                        
2025-07-17T08:42:55.1147686Z         y                        1024(0x400)                        
2025-07-17T08:42:55.1147939Z         z                        1024(0x400)                        
2025-07-17T08:42:55.1148228Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1148503Z       Grid Max Size per Dimension:
2025-07-17T08:42:55.1148741Z         x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1149005Z         y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1149259Z         z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1149550Z       FBarrier Max Size:       32                                 
2025-07-17T08:42:55.1149820Z *******                  
2025-07-17T08:42:55.1150010Z Agent 8                  
2025-07-17T08:42:55.1150332Z *******                  
2025-07-17T08:42:55.1150556Z   Name:                    gfx90a                             
2025-07-17T08:42:55.1150978Z   Uuid:                    GPU-e97bf651691dab9b               
2025-07-17T08:42:55.1151305Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:42:55.1151627Z   Vendor Name:             AMD                                
2025-07-17T08:42:55.1151932Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:42:55.1152241Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:42:55.1152548Z   Float Round Mode:        NEAR                               
2025-07-17T08:42:55.1152878Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:42:55.1153177Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:42:55.1153480Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:42:55.1153779Z   Queue Type:              MULTI                              
2025-07-17T08:42:55.1154066Z   Node:                    7                                  
2025-07-17T08:42:55.1154361Z   Device Type:             GPU                                
2025-07-17T08:42:55.1154626Z   Cache Info:              
2025-07-17T08:42:55.1154844Z     L1:                      16(0x10) KB                        
2025-07-17T08:42:55.1155104Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:42:55.1155374Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:42:55.1155671Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:42:55.1155981Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:42:55.1156296Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:42:55.1156590Z   BDFID:                   45824                              
2025-07-17T08:42:55.1156882Z   Internal Node ID:        7                                  
2025-07-17T08:42:55.1157194Z   Compute Unit:            104                                
2025-07-17T08:42:55.1157500Z   SIMDs per CU:            4                                  
2025-07-17T08:42:55.1157805Z   Shader Engines:          8                                  
2025-07-17T08:42:55.1158118Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:42:55.1158443Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:42:55.1158763Z   Coherent Host Access:    FALSE                              
2025-07-17T08:42:55.1159048Z   Memory Properties:       
2025-07-17T08:42:55.1159270Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:42:55.1159563Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:42:55.1159877Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:42:55.1160197Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1160493Z   Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1160738Z     x                        1024(0x400)                        
2025-07-17T08:42:55.1160999Z     y                        1024(0x400)                        
2025-07-17T08:42:55.1161243Z     z                        1024(0x400)                        
2025-07-17T08:42:55.1161529Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:42:55.1161842Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:42:55.1162158Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1162437Z   Grid Max Size per Dimension:
2025-07-17T08:42:55.1162653Z     x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1162923Z     y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1163184Z     z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1163479Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:42:55.1163951Z   Packet Processor uCode:: 92                                 
2025-07-17T08:42:55.1164278Z   SDMA engine uCode::      9                                  
2025-07-17T08:42:55.1164708Z   IOMMU Support::          None                               
2025-07-17T08:42:55.1164992Z   Pool Info:               
2025-07-17T08:42:55.1165194Z     Pool 1                   
2025-07-17T08:42:55.1165444Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:42:55.1165753Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1166051Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1166372Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1166711Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1167049Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1167383Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1167663Z     Pool 2                   
2025-07-17T08:42:55.1167914Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:42:55.1168223Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1168526Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1168851Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1169209Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1169559Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1169895Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1170201Z     Pool 3                   
2025-07-17T08:42:55.1170449Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:42:55.1170761Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1171062Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1171389Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1171738Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1172085Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1172424Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1172700Z     Pool 4                   
2025-07-17T08:42:55.1172950Z       Segment:                 GROUP                              
2025-07-17T08:42:55.1173239Z       Size:                    64(0x40) KB                        
2025-07-17T08:42:55.1173542Z       Allocatable:             FALSE                              
2025-07-17T08:42:55.1173867Z       Alloc Granule:           0KB                                
2025-07-17T08:42:55.1174202Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:42:55.1174551Z       Alloc Alignment:         0KB                                
2025-07-17T08:42:55.1174870Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1175162Z   ISA Info:                
2025-07-17T08:42:55.1175355Z     ISA 1                    
2025-07-17T08:42:55.1175617Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:42:55.1176028Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:42:55.1176363Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:42:55.1176704Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1177048Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1177370Z       Fast f16:                TRUE                               
2025-07-17T08:42:55.1177684Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1177988Z       Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1178428Z         x                        1024(0x400)                        
2025-07-17T08:42:55.1178835Z         y                        1024(0x400)                        
2025-07-17T08:42:55.1179117Z         z                        1024(0x400)                        
2025-07-17T08:42:55.1179408Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1179710Z       Grid Max Size per Dimension:
2025-07-17T08:42:55.1179963Z         x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1180238Z         y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1180508Z         z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1180813Z       FBarrier Max Size:       32                                 
2025-07-17T08:42:55.1181095Z *******                  
2025-07-17T08:42:55.1181293Z Agent 9                  
2025-07-17T08:42:55.1181486Z *******                  
2025-07-17T08:42:55.1181718Z   Name:                    gfx90a                             
2025-07-17T08:42:55.1182044Z   Uuid:                    GPU-272948773bb8e400               
2025-07-17T08:42:55.1182371Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:42:55.1182709Z   Vendor Name:             AMD                                
2025-07-17T08:42:55.1183034Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:42:55.1183350Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:42:55.1183681Z   Float Round Mode:        NEAR                               
2025-07-17T08:42:55.1184004Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:42:55.1184333Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:42:55.1184644Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:42:55.1184964Z   Queue Type:              MULTI                              
2025-07-17T08:42:55.1185275Z   Node:                    8                                  
2025-07-17T08:42:55.1185573Z   Device Type:             GPU                                
2025-07-17T08:42:55.1185863Z   Cache Info:              
2025-07-17T08:42:55.1186093Z     L1:                      16(0x10) KB                        
2025-07-17T08:42:55.1186380Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:42:55.1186659Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:42:55.1186977Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:42:55.1187293Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:42:55.1187617Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:42:55.1187911Z   BDFID:                   36352                              
2025-07-17T08:42:55.1188203Z   Internal Node ID:        8                                  
2025-07-17T08:42:55.1188527Z   Compute Unit:            104                                
2025-07-17T08:42:55.1188835Z   SIMDs per CU:            4                                  
2025-07-17T08:42:55.1189157Z   Shader Engines:          8                                  
2025-07-17T08:42:55.1189475Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:42:55.1189807Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:42:55.1190149Z   Coherent Host Access:    FALSE                              
2025-07-17T08:42:55.1190440Z   Memory Properties:       
2025-07-17T08:42:55.1190676Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:42:55.1190966Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:42:55.1191298Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:42:55.1191618Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1191921Z   Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1192176Z     x                        1024(0x400)                        
2025-07-17T08:42:55.1192572Z     y                        1024(0x400)                        
2025-07-17T08:42:55.1192963Z     z                        1024(0x400)                        
2025-07-17T08:42:55.1193265Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:42:55.1193593Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:42:55.1193912Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1194214Z   Grid Max Size per Dimension:
2025-07-17T08:42:55.1194445Z     x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1194715Z     y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1194989Z     z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1195293Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:42:55.1195647Z   Packet Processor uCode:: 92                                 
2025-07-17T08:42:55.1195990Z   SDMA engine uCode::      9                                  
2025-07-17T08:42:55.1196326Z   IOMMU Support::          None                               
2025-07-17T08:42:55.1196614Z   Pool Info:               
2025-07-17T08:42:55.1196826Z     Pool 1                   
2025-07-17T08:42:55.1197102Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:42:55.1197417Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1197741Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1198066Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1198423Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1198772Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1199119Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1199425Z     Pool 2                   
2025-07-17T08:42:55.1199696Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:42:55.1200027Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1200334Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1200676Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1201013Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1201376Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1201711Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1201995Z     Pool 3                   
2025-07-17T08:42:55.1202301Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:42:55.1202613Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1202919Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1203263Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1203605Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1203953Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1204289Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1204569Z     Pool 4                   
2025-07-17T08:42:55.1204819Z       Segment:                 GROUP                              
2025-07-17T08:42:55.1205112Z       Size:                    64(0x40) KB                        
2025-07-17T08:42:55.1205422Z       Allocatable:             FALSE                              
2025-07-17T08:42:55.1205742Z       Alloc Granule:           0KB                                
2025-07-17T08:42:55.1206085Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:42:55.1206425Z       Alloc Alignment:         0KB                                
2025-07-17T08:42:55.1206917Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1207217Z   ISA Info:                
2025-07-17T08:42:55.1207537Z     ISA 1                    
2025-07-17T08:42:55.1207815Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:42:55.1208165Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:42:55.1208510Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:42:55.1208858Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1209202Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1209540Z       Fast f16:                TRUE                               
2025-07-17T08:42:55.1209854Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1210170Z       Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1210439Z         x                        1024(0x400)                        
2025-07-17T08:42:55.1210729Z         y                        1024(0x400)                        
2025-07-17T08:42:55.1211007Z         z                        1024(0x400)                        
2025-07-17T08:42:55.1211298Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1211618Z       Grid Max Size per Dimension:
2025-07-17T08:42:55.1211865Z         x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1212142Z         y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1212401Z         z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1212702Z       FBarrier Max Size:       32                                 
2025-07-17T08:42:55.1212981Z *******                  
2025-07-17T08:42:55.1213172Z Agent 10                 
2025-07-17T08:42:55.1213371Z *******                  
2025-07-17T08:42:55.1213585Z   Name:                    gfx90a                             
2025-07-17T08:42:55.1213890Z   Uuid:                    GPU-f3cd2202cbe079af               
2025-07-17T08:42:55.1214211Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:42:55.1214546Z   Vendor Name:             AMD                                
2025-07-17T08:42:55.1214856Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:42:55.1215173Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:42:55.1215497Z   Float Round Mode:        NEAR                               
2025-07-17T08:42:55.1215818Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:42:55.1216225Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:42:55.1216532Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:42:55.1216854Z   Queue Type:              MULTI                              
2025-07-17T08:42:55.1217147Z   Node:                    9                                  
2025-07-17T08:42:55.1217460Z   Device Type:             GPU                                
2025-07-17T08:42:55.1217745Z   Cache Info:              
2025-07-17T08:42:55.1217969Z     L1:                      16(0x10) KB                        
2025-07-17T08:42:55.1218247Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:42:55.1218516Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:42:55.1218817Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:42:55.1219132Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:42:55.1219459Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:42:55.1219767Z   BDFID:                   37632                              
2025-07-17T08:42:55.1220063Z   Internal Node ID:        9                                  
2025-07-17T08:42:55.1220386Z   Compute Unit:            104                                
2025-07-17T08:42:55.1220693Z   SIMDs per CU:            4                                  
2025-07-17T08:42:55.1221169Z   Shader Engines:          8                                  
2025-07-17T08:42:55.1221626Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:42:55.1221978Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:42:55.1222307Z   Coherent Host Access:    FALSE                              
2025-07-17T08:42:55.1222667Z   Memory Properties:       
2025-07-17T08:42:55.1222910Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:42:55.1223201Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:42:55.1223550Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:42:55.1223876Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1224180Z   Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1224428Z     x                        1024(0x400)                        
2025-07-17T08:42:55.1224699Z     y                        1024(0x400)                        
2025-07-17T08:42:55.1224967Z     z                        1024(0x400)                        
2025-07-17T08:42:55.1225253Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:42:55.1225576Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:42:55.1225899Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1226194Z   Grid Max Size per Dimension:
2025-07-17T08:42:55.1226432Z     x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1226711Z     y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1226982Z     z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1227289Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:42:55.1227651Z   Packet Processor uCode:: 92                                 
2025-07-17T08:42:55.1227983Z   SDMA engine uCode::      9                                  
2025-07-17T08:42:55.1228316Z   IOMMU Support::          None                               
2025-07-17T08:42:55.1228595Z   Pool Info:               
2025-07-17T08:42:55.1228811Z     Pool 1                   
2025-07-17T08:42:55.1229079Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:42:55.1229396Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1229706Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1230032Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1230378Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1230723Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1231060Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1231350Z     Pool 2                   
2025-07-17T08:42:55.1231607Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:42:55.1231930Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1232238Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1232576Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1232918Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1233258Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1233587Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1233871Z     Pool 3                   
2025-07-17T08:42:55.1234140Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:42:55.1234438Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:42:55.1234747Z       Allocatable:             TRUE                               
2025-07-17T08:42:55.1235072Z       Alloc Granule:           4KB                                
2025-07-17T08:42:55.1235552Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:42:55.1236002Z       Alloc Alignment:         4KB                                
2025-07-17T08:42:55.1236343Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1236634Z     Pool 4                   
2025-07-17T08:42:55.1236874Z       Segment:                 GROUP                              
2025-07-17T08:42:55.1237183Z       Size:                    64(0x40) KB                        
2025-07-17T08:42:55.1237489Z       Allocatable:             FALSE                              
2025-07-17T08:42:55.1237816Z       Alloc Granule:           0KB                                
2025-07-17T08:42:55.1238152Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:42:55.1238491Z       Alloc Alignment:         0KB                                
2025-07-17T08:42:55.1238829Z       Accessible by all:       FALSE                              
2025-07-17T08:42:55.1239116Z   ISA Info:                
2025-07-17T08:42:55.1239326Z     ISA 1                    
2025-07-17T08:42:55.1239596Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:42:55.1239944Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:42:55.1240275Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:42:55.1240620Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1240963Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:42:55.1241273Z       Fast f16:                TRUE                               
2025-07-17T08:42:55.1241587Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:42:55.1241887Z       Workgroup Max Size per Dimension:
2025-07-17T08:42:55.1242149Z         x                        1024(0x400)                        
2025-07-17T08:42:55.1242410Z         y                        1024(0x400)                        
2025-07-17T08:42:55.1242680Z         z                        1024(0x400)                        
2025-07-17T08:42:55.1242980Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:42:55.1243271Z       Grid Max Size per Dimension:
2025-07-17T08:42:55.1243519Z         x                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1243786Z         y                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1244063Z         z                        4294967295(0xffffffff)             
2025-07-17T08:42:55.1244360Z       FBarrier Max Size:       32                                 
2025-07-17T08:42:55.1244652Z *** Done ***             
2025-07-17T08:42:55.1263760Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-07-17T08:42:55.1264148Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-07-17T08:42:55.1264770Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-07-17T08:42:55.1265364Z if [[ $ngpu -eq 0 ]]; then
2025-07-17T08:42:55.1265685Z     echo "Error: Failed to detect any GPUs on the runner"
2025-07-17T08:42:55.1266008Z     echo "$msg"
2025-07-17T08:42:55.1266207Z     exit 1
2025-07-17T08:42:55.1266392Z fi
2025-07-17T08:42:55.1266575Z if [[ $ngpu -eq 1 ]]; then
2025-07-17T08:42:55.1266953Z     echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs"
2025-07-17T08:42:55.1267326Z     echo "$msg"
2025-07-17T08:42:55.1267525Z     exit 1
2025-07-17T08:42:55.1267695Z fi
2025-07-17T08:42:55.1291416Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:55.1291736Z env:
2025-07-17T08:42:55.1291910Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:55.1292119Z ##[endgroup]
2025-07-17T08:42:55.2878734Z ##[group]Run pytorch/pytorch/.github/actions/diskspace-cleanup@main
2025-07-17T08:42:55.2879286Z with:
2025-07-17T08:42:55.2879477Z   diskspace-cutoff: 70
2025-07-17T08:42:55.2879691Z env:
2025-07-17T08:42:55.2880023Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:55.2880257Z ##[endgroup]
2025-07-17T08:42:55.2934384Z ##[group]Run set -ex
2025-07-17T08:42:55.2934634Z set -ex
2025-07-17T08:42:55.2934829Z diskspace_cutoff=70
2025-07-17T08:42:55.2935136Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2025-07-17T08:42:55.2935481Z if [ ! -d "$docker_root_dir" ]; then
2025-07-17T08:42:55.2936005Z     echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check."
2025-07-17T08:42:55.2936429Z     exit 0
2025-07-17T08:42:55.2936607Z fi
2025-07-17T08:42:55.2936962Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-07-17T08:42:55.2937702Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-07-17T08:42:55.2938344Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2025-07-17T08:42:55.2938652Z     docker system prune -af
2025-07-17T08:42:55.2939058Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-07-17T08:42:55.2939520Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2025-07-17T08:42:55.2940002Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2025-07-17T08:42:55.2940436Z         echo "$msg"
2025-07-17T08:42:55.2940645Z         exit 1
2025-07-17T08:42:55.2940827Z     else
2025-07-17T08:42:55.2941050Z         difference=$((diskspace - diskspace_new))
2025-07-17T08:42:55.2941375Z         echo "Diskspace saved: $difference percent"
2025-07-17T08:42:55.2941645Z     fi
2025-07-17T08:42:55.2941806Z fi
2025-07-17T08:42:55.2964041Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:55.2964360Z env:
2025-07-17T08:42:55.2964535Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:55.2964743Z ##[endgroup]
2025-07-17T08:42:55.3011233Z + diskspace_cutoff=70
2025-07-17T08:42:55.3015885Z ++ docker info -f '{{.DockerRootDir}}'
2025-07-17T08:42:55.3609766Z + docker_root_dir=/media/4TB/docker-rootless
2025-07-17T08:42:55.3610327Z + '[' '!' -d /media/4TB/docker-rootless ']'
2025-07-17T08:42:55.3622136Z ++ df -H --output=pcent /media/4TB/docker-rootless
2025-07-17T08:42:55.3623965Z ++ sed -n 2p
2025-07-17T08:42:55.3625656Z ++ sed s/%//
2025-07-17T08:42:55.3628312Z ++ sed 's/ //'
2025-07-17T08:42:55.3650336Z + diskspace=17
2025-07-17T08:42:55.3650925Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2025-07-17T08:42:55.3651539Z + [[ 17 -ge 70 ]]
2025-07-17T08:42:55.3684060Z ##[group]Run RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-07-17T08:42:55.3684473Z RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-07-17T08:42:55.3684813Z rm -rf "${RUNNER_ARTIFACT_DIR}"
2025-07-17T08:42:55.3685101Z mkdir -p "${RUNNER_ARTIFACT_DIR}"
2025-07-17T08:42:55.3685475Z echo "RUNNER_ARTIFACT_DIR=${RUNNER_ARTIFACT_DIR}" >> "${GITHUB_ENV}"
2025-07-17T08:42:55.3685827Z 
2025-07-17T08:42:55.3686084Z RUNNER_TEST_RESULTS_DIR="${RUNNER_TEMP}/test-results"
2025-07-17T08:42:55.3686422Z rm -rf "${RUNNER_TEST_RESULTS_DIR}"
2025-07-17T08:42:55.3686718Z mkdir -p "${RUNNER_TEST_RESULTS_DIR}"
2025-07-17T08:42:55.3687118Z echo "RUNNER_TEST_RESULTS_DIR=${RUNNER_TEST_RESULTS_DIR}" >> "${GITHUB_ENV}"
2025-07-17T08:42:55.3687483Z 
2025-07-17T08:42:55.3687683Z RUNNER_DOCS_DIR="${RUNNER_TEMP}/docs"
2025-07-17T08:42:55.3687954Z rm -rf "${RUNNER_DOCS_DIR}"
2025-07-17T08:42:55.3688439Z mkdir -p "${RUNNER_DOCS_DIR}"
2025-07-17T08:42:55.3688761Z echo "RUNNER_DOCS_DIR=${RUNNER_DOCS_DIR}" >> "${GITHUB_ENV}"
2025-07-17T08:42:55.3713352Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:55.3713683Z env:
2025-07-17T08:42:55.3713860Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:55.3714080Z ##[endgroup]
2025-07-17T08:42:55.3893738Z ##[group]Run env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-07-17T08:42:55.3894214Z env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-07-17T08:42:55.3894636Z env | grep '^CI' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-07-17T08:42:55.3918305Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:55.3918619Z env:
2025-07-17T08:42:55.3918805Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:55.3919155Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:42:55.3919697Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:42:55.3920187Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:42:55.3920529Z ##[endgroup]
2025-07-17T08:42:55.4033396Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-07-17T08:42:55.4034002Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-07-17T08:42:55.4034436Z # Add render group for container creation.
2025-07-17T08:42:55.4034789Z render_gid=`cat /etc/group | grep render | cut -d: -f3`
2025-07-17T08:42:55.4035217Z # Ensure GPU isolation if pod is part of kubernetes setup with DEVICE_FLAG.
2025-07-17T08:42:55.4035645Z if [ -f "/etc/podinfo/gha-render-devices" ]; then
2025-07-17T08:42:55.4036005Z   DEVICE_FLAG=$(cat /etc/podinfo/gha-render-devices)
2025-07-17T08:42:55.4036296Z else
2025-07-17T08:42:55.4036514Z   DEVICE_FLAG="--device /dev/dri"
2025-07-17T08:42:55.4036752Z fi
2025-07-17T08:42:55.4037150Z # The --group-add daemon and --group-add bin are needed in the Ubuntu 24.04 and Almalinux OSs respectively.
2025-07-17T08:42:55.4037759Z # This is due to the device files (/dev/kfd & /dev/dri) being owned by video group on bare metal.
2025-07-17T08:42:55.4038315Z # This video group ID maps to subgid 1 inside the docker image due to the /etc/subgid entries.
2025-07-17T08:42:55.4038894Z # The group name corresponding to group ID 1 can change depending on the OS, so both are necessary.
2025-07-17T08:42:55.4039894Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd $DEVICE_FLAG --group-add video --group-add $render_gid --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host" >> "${GITHUB_ENV}"
2025-07-17T08:42:55.4063245Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:55.4063567Z env:
2025-07-17T08:42:55.4063736Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:55.4064092Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:42:55.4064609Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:42:55.4065108Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:42:55.4065429Z ##[endgroup]
2025-07-17T08:42:55.4205575Z ##[group]Run aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722
2025-07-17T08:42:55.4206063Z with:
2025-07-17T08:42:55.4206376Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only
2025-07-17T08:42:55.4206756Z   aws-region: us-east-1
2025-07-17T08:42:55.4206983Z   role-duration-seconds: 18000
2025-07-17T08:42:55.4207242Z   audience: sts.amazonaws.com
2025-07-17T08:42:55.4207467Z env:
2025-07-17T08:42:55.4207650Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:55.4208005Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:42:55.4208746Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:42:55.4209244Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:42:55.4210097Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:42:55.4210809Z ##[endgroup]
2025-07-17T08:42:55.7283065Z Assuming role with OIDC
2025-07-17T08:42:55.9128892Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions
2025-07-17T08:42:55.9748915Z ##[group]Run aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076
2025-07-17T08:42:55.9749378Z with:
2025-07-17T08:42:55.9749591Z   mask-password: true
2025-07-17T08:42:55.9749828Z   registry-type: private
2025-07-17T08:42:55.9750048Z   skip-logout: false
2025-07-17T08:42:55.9750267Z env:
2025-07-17T08:42:55.9750459Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:55.9750836Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:42:55.9751377Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:42:55.9751892Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:42:55.9752737Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:42:55.9753490Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:42:55.9753764Z   AWS_REGION: us-east-1
2025-07-17T08:42:55.9754279Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:42:55.9754616Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:42:55.9759893Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:42:55.9760123Z ##[endgroup]
2025-07-17T08:42:56.3298359Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:56.6841715Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2025-07-17T08:42:56.6842154Z with:
2025-07-17T08:42:56.6842779Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:56.6843471Z   use-custom-docker-registry: true
2025-07-17T08:42:56.6843742Z   docker-build-dir: .ci/docker
2025-07-17T08:42:56.6844009Z   docker-build-script: ./build.sh
2025-07-17T08:42:56.6844264Z   working-directory: .
2025-07-17T08:42:56.6844573Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:56.6844917Z   force-push: false
2025-07-17T08:42:56.6845127Z env:
2025-07-17T08:42:56.6845321Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:56.6845698Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:42:56.6846238Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:42:56.6846787Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:42:56.6847651Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:42:56.6848428Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:42:56.6848670Z   AWS_REGION: us-east-1
2025-07-17T08:42:56.6849021Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:42:56.6849358Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:42:56.6854615Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:42:56.6854839Z ##[endgroup]
2025-07-17T08:42:56.6897617Z ##[group]Run set -ex
2025-07-17T08:42:56.6897889Z set -ex
2025-07-17T08:42:56.6898097Z 
2025-07-17T08:42:56.6898457Z # If the docker build directory or the build script doesn't exist, the action will
2025-07-17T08:42:56.6899040Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2025-07-17T08:42:56.6899751Z # job could then download the pre-built image as usual
2025-07-17T08:42:56.6900340Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then
2025-07-17T08:42:56.6900890Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2025-07-17T08:42:56.6901181Z else
2025-07-17T08:42:56.6901417Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2025-07-17T08:42:56.6901801Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-07-17T08:42:56.6902147Z 
2025-07-17T08:42:56.6902625Z   echo "Not using custom ECR registry.  Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..."
2025-07-17T08:42:56.6903171Z   exit 0
2025-07-17T08:42:56.6903364Z fi
2025-07-17T08:42:56.6903555Z 
2025-07-17T08:42:56.6903849Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2025-07-17T08:42:56.6904368Z   # The docker image name already includes the ECR prefix and tag, so we can just
2025-07-17T08:42:56.6904817Z   # use it as it is, but first let's extract the tag
2025-07-17T08:42:56.6905222Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2025-07-17T08:42:56.6905654Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-07-17T08:42:56.6906064Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-07-17T08:42:56.6906400Z else
2025-07-17T08:42:56.6906627Z   if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then
2025-07-17T08:42:56.6906951Z     CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:}
2025-07-17T08:42:56.6907281Z     DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*}
2025-07-17T08:42:56.6907558Z   fi
2025-07-17T08:42:56.6908166Z   DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2025-07-17T08:42:56.6908699Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-07-17T08:42:56.6909227Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-07-17T08:42:56.6909803Z   echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}"
2025-07-17T08:42:56.6910152Z fi
2025-07-17T08:42:56.6936559Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:56.6936901Z env:
2025-07-17T08:42:56.6937101Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:56.6937473Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:42:56.6938019Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:42:56.6938537Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:42:56.6939379Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:42:56.6940133Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:42:56.6940394Z   AWS_REGION: us-east-1
2025-07-17T08:42:56.6940702Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:42:56.6941025Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:42:56.6946351Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:42:56.6946584Z   REPO_NAME: pytorch
2025-07-17T08:42:56.6947197Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:56.6947877Z   DOCKER_BUILD_DIR: .ci/docker
2025-07-17T08:42:56.6948126Z   DOCKER_BUILD_SCRIPT: ./build.sh
2025-07-17T08:42:56.6948450Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:56.6948802Z   USE_CUSTOM_DOCKER_REGISTRY: true
2025-07-17T08:42:56.6949060Z   CUSTOM_TAG_PREFIX: 
2025-07-17T08:42:56.6949478Z ##[endgroup]
2025-07-17T08:42:56.6996270Z + [[ -d .ci/docker ]]
2025-07-17T08:42:56.6996524Z + [[ -f .ci/docker/./build.sh ]]
2025-07-17T08:42:56.6996777Z + [[ true == \t\r\u\e ]]
2025-07-17T08:42:56.6996994Z + echo skip=false
2025-07-17T08:42:56.7002739Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2025-07-17T08:42:56.7011153Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:56.7012537Z ++ awk -F '[:,]' '{print $2}'
2025-07-17T08:42:56.7043087Z + DOCKER_TAG=pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:56.7044181Z + echo docker-tag=pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:56.7045676Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:56.7080595Z ##[group]Run set +e
2025-07-17T08:42:56.7080886Z set +e
2025-07-17T08:42:56.7081099Z set -x
2025-07-17T08:42:56.7081298Z 
2025-07-17T08:42:56.7081494Z login() {
2025-07-17T08:42:56.7081916Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-07-17T08:42:56.7082375Z }
2025-07-17T08:42:56.7082567Z 
2025-07-17T08:42:56.7082753Z retry () {
2025-07-17T08:42:56.7083000Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-07-17T08:42:56.7083302Z }
2025-07-17T08:42:56.7083512Z 
2025-07-17T08:42:56.7083731Z retry login "${DOCKER_REGISTRY}"
2025-07-17T08:42:56.7084020Z 
2025-07-17T08:42:56.7084257Z START_TIME=$(date +%s)
2025-07-17T08:42:56.7084823Z # Wait up to 120 minutes
2025-07-17T08:42:56.7085202Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do
2025-07-17T08:42:56.7085654Z   # Check if image already exists, if it does then skip building it
2025-07-17T08:42:56.7086115Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2025-07-17T08:42:56.7086424Z     exit 0
2025-07-17T08:42:56.7086642Z   fi
2025-07-17T08:42:56.7086845Z 
2025-07-17T08:42:56.7087199Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2025-07-17T08:42:56.7087773Z   # use this to differentiate between the Docker build and regular build jobs. For the
2025-07-17T08:42:56.7088321Z   # latter, it will wait for the Docker images to become available before continuing
2025-07-17T08:42:56.7088760Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2025-07-17T08:42:56.7089112Z     # It's a Docker build job, let's build the image
2025-07-17T08:42:56.7089443Z     break
2025-07-17T08:42:56.7089666Z   else
2025-07-17T08:42:56.7089960Z     # It's a regular build job, wait for the image to become available
2025-07-17T08:42:56.7090327Z     sleep 300
2025-07-17T08:42:56.7090553Z   fi
2025-07-17T08:42:56.7090753Z done
2025-07-17T08:42:56.7090943Z 
2025-07-17T08:42:56.7091248Z # NB: This part requires a full checkout. Otherwise, the merge base will
2025-07-17T08:42:56.7091731Z # be empty.  The default action would be to continue rebuild the image
2025-07-17T08:42:56.7092174Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2025-07-17T08:42:56.7092579Z   # if we're on the base branch then use the parent commit
2025-07-17T08:42:56.7092942Z   MERGE_BASE=$(git rev-parse HEAD~)
2025-07-17T08:42:56.7093220Z else
2025-07-17T08:42:56.7093510Z   # otherwise we're on a PR, so use the most recent base commit
2025-07-17T08:42:56.7094129Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2025-07-17T08:42:56.7094443Z fi
2025-07-17T08:42:56.7094633Z 
2025-07-17T08:42:56.7094840Z if [[ -z "${MERGE_BASE}" ]]; then
2025-07-17T08:42:56.7095146Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-07-17T08:42:56.7095420Z 
2025-07-17T08:42:56.7095811Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2025-07-17T08:42:56.7096377Z   exit 0
2025-07-17T08:42:56.7096584Z fi
2025-07-17T08:42:56.7096773Z 
2025-07-17T08:42:56.7097041Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2025-07-17T08:42:56.7097611Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2025-07-17T08:42:56.7098103Z   exit 1
2025-07-17T08:42:56.7098310Z fi
2025-07-17T08:42:56.7098499Z 
2025-07-17T08:42:56.7098810Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2025-07-17T08:42:56.7099361Z # If no image exists but the hash is the same as the previous hash then we should error out here
2025-07-17T08:42:56.7099853Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2025-07-17T08:42:56.7100428Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2025-07-17T08:42:56.7101061Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2025-07-17T08:42:56.7101443Z fi
2025-07-17T08:42:56.7101619Z 
2025-07-17T08:42:56.7101846Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-07-17T08:42:56.7128358Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:56.7128711Z env:
2025-07-17T08:42:56.7128909Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:56.7129495Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:42:56.7130050Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:42:56.7130557Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:42:56.7131398Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:42:56.7132157Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:42:56.7132402Z   AWS_REGION: us-east-1
2025-07-17T08:42:56.7132783Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:42:56.7133124Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:42:56.7138462Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:42:56.7138717Z   DOCKER_BUILD_DIR: .ci/docker
2025-07-17T08:42:56.7139012Z   BASE_REVISION: a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:42:56.7139718Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:56.7140538Z   DOCKER_TAG: pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:56.7141044Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:56.7141370Z   DOCKER_PUSH: 
2025-07-17T08:42:56.7141581Z ##[endgroup]
2025-07-17T08:42:56.7191658Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:56.7192070Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:56.7196899Z + aws ecr get-login-password --region us-east-1
2025-07-17T08:42:56.7198638Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:57.7419912Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2025-07-17T08:42:57.7420858Z Configure a credential helper to remove this warning. See
2025-07-17T08:42:57.7422417Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2025-07-17T08:42:57.7423418Z 
2025-07-17T08:42:57.7428653Z Login Succeeded
2025-07-17T08:42:57.7467503Z ++ date +%s
2025-07-17T08:42:57.7483873Z + START_TIME=1752741777
2025-07-17T08:42:57.7490566Z ++ date +%s
2025-07-17T08:42:57.7506166Z + [[ 1752734577 -lt 1752741777 ]]
2025-07-17T08:42:57.7506965Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:58.4514060Z {
2025-07-17T08:42:58.4514453Z 	"schemaVersion": 2,
2025-07-17T08:42:58.4515115Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2025-07-17T08:42:58.4515843Z 	"config": {
2025-07-17T08:42:58.4516423Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2025-07-17T08:42:58.4517113Z 		"size": 28418,
2025-07-17T08:42:58.4517831Z 		"digest": "sha256:0f464723fd0e297450625b7697dd0dc4ef837960dceaa5bd0f2f4d7d8b9987eb"
2025-07-17T08:42:58.4518673Z 	},
2025-07-17T08:42:58.4519021Z 	"layers": [
2025-07-17T08:42:58.4519386Z 		{
2025-07-17T08:42:58.4520080Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4520689Z 			"size": 30446997,
2025-07-17T08:42:58.4521289Z 			"digest": "sha256:66587c81b81a58d07e40c48d900a1517516bbf58c4378c687d89d645824f5e5f"
2025-07-17T08:42:58.4521955Z 		},
2025-07-17T08:42:58.4522245Z 		{
2025-07-17T08:42:58.4522725Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4523349Z 			"size": 1554,
2025-07-17T08:42:58.4523966Z 			"digest": "sha256:a3c451a3328b650c5ddade4025cc44a20eea3fa108daf68fb805d5038f4b327a"
2025-07-17T08:42:58.4524647Z 		},
2025-07-17T08:42:58.4524939Z 		{
2025-07-17T08:42:58.4525417Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4526074Z 			"size": 313270340,
2025-07-17T08:42:58.4526610Z 			"digest": "sha256:74e64bc968c1cf8f97d841b7ea0266de87fa2e1b8b4254f71e0350252bcc7f1a"
2025-07-17T08:42:58.4527307Z 		},
2025-07-17T08:42:58.4527505Z 		{
2025-07-17T08:42:58.4527817Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4528195Z 			"size": 703,
2025-07-17T08:42:58.4528583Z 			"digest": "sha256:9bcb8a79b45b53f1f900b5be887bf24d1166b93e73447e1ed1d7d17265eec376"
2025-07-17T08:42:58.4529018Z 		},
2025-07-17T08:42:58.4529205Z 		{
2025-07-17T08:42:58.4529507Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4529891Z 			"size": 1212,
2025-07-17T08:42:58.4530272Z 			"digest": "sha256:35c764410de1133f1414d9b3f52ac20cc4a04bc5dad2555fce5fed7fc7b53e27"
2025-07-17T08:42:58.4530708Z 		},
2025-07-17T08:42:58.4530887Z 		{
2025-07-17T08:42:58.4531181Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4531558Z 			"size": 485,
2025-07-17T08:42:58.4531923Z 			"digest": "sha256:501f1b9049b569de8832d393bd28cb9461fb1a5748f254493059d06f95078d42"
2025-07-17T08:42:58.4532356Z 		},
2025-07-17T08:42:58.4532538Z 		{
2025-07-17T08:42:58.4532833Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4533218Z 			"size": 110343714,
2025-07-17T08:42:58.4533608Z 			"digest": "sha256:52662c4baedc58d84a36538f1ec46a58ef357581acf221294a5b5400df3613a7"
2025-07-17T08:42:58.4534043Z 		},
2025-07-17T08:42:58.4534232Z 		{
2025-07-17T08:42:58.4534543Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4534933Z 			"size": 4084,
2025-07-17T08:42:58.4535314Z 			"digest": "sha256:e176b5597377db9ba0da10a2949417bd291ab8c8880d4cf810bfa46963514355"
2025-07-17T08:42:58.4535757Z 		},
2025-07-17T08:42:58.4536241Z 		{
2025-07-17T08:42:58.4536560Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4536944Z 			"size": 1709,
2025-07-17T08:42:58.4537294Z 			"digest": "sha256:0d668fce00b22755f15b2c6a7185f5e4cd9e016804fd763b9c213c24330ad63e"
2025-07-17T08:42:58.4537681Z 		},
2025-07-17T08:42:58.4538020Z 		{
2025-07-17T08:42:58.4538302Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4538647Z 			"size": 724,
2025-07-17T08:42:58.4538984Z 			"digest": "sha256:d81f985aa40d5111b424e8084dfa90790c3078ab270debfa5b46f37c2423c5ce"
2025-07-17T08:42:58.4539366Z 		},
2025-07-17T08:42:58.4539531Z 		{
2025-07-17T08:42:58.4539795Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4540141Z 			"size": 3310464312,
2025-07-17T08:42:58.4540501Z 			"digest": "sha256:ec9c560bb82fd80fe1a703d8635753191c3bf91ce72bf3ef3d7c988cdc60709f"
2025-07-17T08:42:58.4540888Z 		},
2025-07-17T08:42:58.4541047Z 		{
2025-07-17T08:42:58.4541311Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4541649Z 			"size": 381,
2025-07-17T08:42:58.4541982Z 			"digest": "sha256:bef037cce55fd4003ac13320e72dc1e8d781127e52efd4608ef8cc5f7d5a4898"
2025-07-17T08:42:58.4542371Z 		},
2025-07-17T08:42:58.4542535Z 		{
2025-07-17T08:42:58.4542811Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4543148Z 			"size": 233575,
2025-07-17T08:42:58.4543490Z 			"digest": "sha256:cc14155fac7b29e7b199a85aef3d20af7112604a2dc86d5dd0a4c89f269d5457"
2025-07-17T08:42:58.4543872Z 		},
2025-07-17T08:42:58.4544035Z 		{
2025-07-17T08:42:58.4544303Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4544640Z 			"size": 791,
2025-07-17T08:42:58.4544981Z 			"digest": "sha256:98b40f6827d2be8fb97e8ebec31f1b0211c3efcaf33d3f81ce804245254e0f44"
2025-07-17T08:42:58.4545362Z 		},
2025-07-17T08:42:58.4545522Z 		{
2025-07-17T08:42:58.4545790Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4546133Z 			"size": 106,
2025-07-17T08:42:58.4546460Z 			"digest": "sha256:2816c98112963b1630214fb5e27007cd06f8d422a17d61f642a0ca764e305c64"
2025-07-17T08:42:58.4546831Z 		},
2025-07-17T08:42:58.4546994Z 		{
2025-07-17T08:42:58.4547267Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4547763Z 			"size": 1495,
2025-07-17T08:42:58.4548119Z 			"digest": "sha256:835aba0facaacf7c3dca5e1f039808b834ec29306e4eabf749c6541fa041ed2d"
2025-07-17T08:42:58.4548510Z 		},
2025-07-17T08:42:58.4548665Z 		{
2025-07-17T08:42:58.4548940Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4549295Z 			"size": 455280682,
2025-07-17T08:42:58.4549650Z 			"digest": "sha256:3a7045d55b23c0af9416a558f55fd6b5b7c56060d2fb34178134bd11197aea01"
2025-07-17T08:42:58.4550038Z 		},
2025-07-17T08:42:58.4550208Z 		{
2025-07-17T08:42:58.4550482Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4550827Z 			"size": 162,
2025-07-17T08:42:58.4551163Z 			"digest": "sha256:23e65526d2c2ad371238cebf590b9c1a5215668bc35de974fb423e3cb6674ecf"
2025-07-17T08:42:58.4551547Z 		},
2025-07-17T08:42:58.4551715Z 		{
2025-07-17T08:42:58.4551972Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4552319Z 			"size": 2370,
2025-07-17T08:42:58.4552662Z 			"digest": "sha256:0bb9b2bb3a77fff33a12e20d8fe9f5c637c339facf000c4407fc1c4515980aa8"
2025-07-17T08:42:58.4553054Z 		},
2025-07-17T08:42:58.4553221Z 		{
2025-07-17T08:42:58.4553489Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4553832Z 			"size": 7164120706,
2025-07-17T08:42:58.4554177Z 			"digest": "sha256:89c2ed775d8d10f463a2ed1ab047b356444911b18742cee8b651d57c2cf42ace"
2025-07-17T08:42:58.4554559Z 		},
2025-07-17T08:42:58.4554719Z 		{
2025-07-17T08:42:58.4554976Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4555304Z 			"size": 105,
2025-07-17T08:42:58.4555635Z 			"digest": "sha256:22367c15654a15dbf35e3b8360202ebac7bdace0703233bf286a9a4f0d726c59"
2025-07-17T08:42:58.4556027Z 		},
2025-07-17T08:42:58.4556196Z 		{
2025-07-17T08:42:58.4556461Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4556800Z 			"size": 611,
2025-07-17T08:42:58.4557284Z 			"digest": "sha256:41dd58eb2af160ecb43af63c54b552825ebfa78a973bd316c44eb1ddfff16d18"
2025-07-17T08:42:58.4557678Z 		},
2025-07-17T08:42:58.4557850Z 		{
2025-07-17T08:42:58.4558114Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4558464Z 			"size": 677677419,
2025-07-17T08:42:58.4559050Z 			"digest": "sha256:0c0bf7dba119dca098a5b3c549795939817b9de42b80ee5dbd9f9d6871cdd5da"
2025-07-17T08:42:58.4559489Z 		},
2025-07-17T08:42:58.4559661Z 		{
2025-07-17T08:42:58.4559933Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4560276Z 			"size": 111,
2025-07-17T08:42:58.4560622Z 			"digest": "sha256:d49b2c2152c7848476fc7c52aad3b33c65554fcaf9f0c72dd790a0c0ac68a000"
2025-07-17T08:42:58.4561012Z 		},
2025-07-17T08:42:58.4561180Z 		{
2025-07-17T08:42:58.4561452Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4561796Z 			"size": 1556,
2025-07-17T08:42:58.4562147Z 			"digest": "sha256:e5f6086238542da1a27d6d92acd14a4320cdca70eedb0b550e883bb0c93d4c7b"
2025-07-17T08:42:58.4562554Z 		},
2025-07-17T08:42:58.4562724Z 		{
2025-07-17T08:42:58.4563005Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4563357Z 			"size": 107,
2025-07-17T08:42:58.4563684Z 			"digest": "sha256:f833b0ca2e595948d8dfd847ff60d2125781be4b1216d293329bfe8fa3f88378"
2025-07-17T08:42:58.4564077Z 		},
2025-07-17T08:42:58.4564244Z 		{
2025-07-17T08:42:58.4564517Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4564864Z 			"size": 166,
2025-07-17T08:42:58.4565202Z 			"digest": "sha256:ab9877515928d6413a2db6769cf16e7bfafa92ab8967349471de8610df1ee728"
2025-07-17T08:42:58.4565587Z 		},
2025-07-17T08:42:58.4565759Z 		{
2025-07-17T08:42:58.4566032Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4566389Z 			"size": 2801144,
2025-07-17T08:42:58.4566734Z 			"digest": "sha256:9f52d3b83f081eb26e37656421cc991942dc5c025ec4ebf8e5312e61f5a9aff2"
2025-07-17T08:42:58.4567253Z 		},
2025-07-17T08:42:58.4567435Z 		{
2025-07-17T08:42:58.4567702Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4568044Z 			"size": 107,
2025-07-17T08:42:58.4568373Z 			"digest": "sha256:a03e9776803ba5945e13460659b97af55674b9246fddce732c65215f14ce6846"
2025-07-17T08:42:58.4568752Z 		},
2025-07-17T08:42:58.4568921Z 		{
2025-07-17T08:42:58.4569195Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4569539Z 			"size": 802,
2025-07-17T08:42:58.4569879Z 			"digest": "sha256:60bb50caf36b82f9e2c5219ce0415da7571e910f43a183d4f07e1b5f8357159b"
2025-07-17T08:42:58.4570269Z 		},
2025-07-17T08:42:58.4570428Z 		{
2025-07-17T08:42:58.4570711Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4571052Z 			"size": 26113574,
2025-07-17T08:42:58.4571403Z 			"digest": "sha256:6fd0a137ed1d7b3abed2c19e7b0013c411d0d0a0642c8a45f7064e704f71c090"
2025-07-17T08:42:58.4571802Z 		},
2025-07-17T08:42:58.4571972Z 		{
2025-07-17T08:42:58.4572247Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4572594Z 			"size": 104,
2025-07-17T08:42:58.4572929Z 			"digest": "sha256:865dca06b4bef4059f415bdfb4c535212687fed6f88d9d4202721c50c4dddf43"
2025-07-17T08:42:58.4573316Z 		},
2025-07-17T08:42:58.4573468Z 		{
2025-07-17T08:42:58.4573731Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4574069Z 			"size": 425,
2025-07-17T08:42:58.4574405Z 			"digest": "sha256:d2b21c379220ce9e7b232a254b4d6a6c8f5bb93b6c4b32d38146b4f7929d83d4"
2025-07-17T08:42:58.4574789Z 		},
2025-07-17T08:42:58.4574952Z 		{
2025-07-17T08:42:58.4575222Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4575560Z 			"size": 19308729,
2025-07-17T08:42:58.4575972Z 			"digest": "sha256:27e06e0e000d0089df33a38f23ba60254494e6a2a47d1a390cf498659527ca4e"
2025-07-17T08:42:58.4576353Z 		},
2025-07-17T08:42:58.4576674Z 		{
2025-07-17T08:42:58.4576939Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4577288Z 			"size": 425,
2025-07-17T08:42:58.4577633Z 			"digest": "sha256:e580816fdea358adca6416fddf604089befb1c5e1ebe89954e79d176c0c88b09"
2025-07-17T08:42:58.4578028Z 		},
2025-07-17T08:42:58.4578195Z 		{
2025-07-17T08:42:58.4578456Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4578798Z 			"size": 724,
2025-07-17T08:42:58.4579130Z 			"digest": "sha256:d81f985aa40d5111b424e8084dfa90790c3078ab270debfa5b46f37c2423c5ce"
2025-07-17T08:42:58.4579514Z 		},
2025-07-17T08:42:58.4579669Z 		{
2025-07-17T08:42:58.4579930Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4580261Z 			"size": 141,
2025-07-17T08:42:58.4580593Z 			"digest": "sha256:42a16ea59e6842dd518eb22d4c479d66b72d202c46406dccb01ea366651dd9cb"
2025-07-17T08:42:58.4580978Z 		},
2025-07-17T08:42:58.4581143Z 		{
2025-07-17T08:42:58.4581428Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4581764Z 			"size": 137,
2025-07-17T08:42:58.4582099Z 			"digest": "sha256:e3bdcc2c9a2c6093950eadf56a935f113e3b08f3d6eaa93890df99276389800d"
2025-07-17T08:42:58.4582507Z 		},
2025-07-17T08:42:58.4582677Z 		{
2025-07-17T08:42:58.4582945Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4583290Z 			"size": 5098803561,
2025-07-17T08:42:58.4583635Z 			"digest": "sha256:79d9977620eecca9d68daa4ada200d465c7873b097633f8f345ffc781c27f898"
2025-07-17T08:42:58.4584021Z 		},
2025-07-17T08:42:58.4584188Z 		{
2025-07-17T08:42:58.4584468Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4584807Z 			"size": 193,
2025-07-17T08:42:58.4585142Z 			"digest": "sha256:9f5a3d4f8993392fe726ea47c2bf39cc61f61256cee93c973fb2cbafc38b7a83"
2025-07-17T08:42:58.4585530Z 		},
2025-07-17T08:42:58.4585696Z 		{
2025-07-17T08:42:58.4585958Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4586449Z 			"size": 346,
2025-07-17T08:42:58.4586787Z 			"digest": "sha256:b23e8f7f8b435c3bde38e1b845ee7dc5126fc5d780a924f94f2e509a3c3d1447"
2025-07-17T08:42:58.4587172Z 		},
2025-07-17T08:42:58.4587333Z 		{
2025-07-17T08:42:58.4587594Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4587928Z 			"size": 88297,
2025-07-17T08:42:58.4588269Z 			"digest": "sha256:ce8be175f8c65e12653647cf4f8422fb1a80cccd4eccd43709601e635f74a7d5"
2025-07-17T08:42:58.4588658Z 		},
2025-07-17T08:42:58.4588822Z 		{
2025-07-17T08:42:58.4589081Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4589426Z 			"size": 106,
2025-07-17T08:42:58.4589748Z 			"digest": "sha256:55fc3af08ef2a75b2a6d92c9471851245d654481b650f63b08c048298af23888"
2025-07-17T08:42:58.4590118Z 		},
2025-07-17T08:42:58.4590281Z 		{
2025-07-17T08:42:58.4590543Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4590881Z 			"size": 1666,
2025-07-17T08:42:58.4591212Z 			"digest": "sha256:496c712257d4af1014200db779525259969ca86594f3fc09da71981884f1688d"
2025-07-17T08:42:58.4591584Z 		},
2025-07-17T08:42:58.4591743Z 		{
2025-07-17T08:42:58.4592006Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4592337Z 			"size": 724,
2025-07-17T08:42:58.4592662Z 			"digest": "sha256:d81f985aa40d5111b424e8084dfa90790c3078ab270debfa5b46f37c2423c5ce"
2025-07-17T08:42:58.4593036Z 		},
2025-07-17T08:42:58.4593188Z 		{
2025-07-17T08:42:58.4593444Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4593780Z 			"size": 137,
2025-07-17T08:42:58.4594099Z 			"digest": "sha256:5277b610601a1d4b1d7671d06690f8c645b403ff8121576e7c9199ecc0aafd81"
2025-07-17T08:42:58.4594467Z 		},
2025-07-17T08:42:58.4594624Z 		{
2025-07-17T08:42:58.4594886Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4595223Z 			"size": 120,
2025-07-17T08:42:58.4595714Z 			"digest": "sha256:cdb570d9a441be069ed53d7dd161902340833096b46d848fe9d2c89424de6b66"
2025-07-17T08:42:58.4596101Z 		},
2025-07-17T08:42:58.4596266Z 		{
2025-07-17T08:42:58.4596519Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4596858Z 			"size": 5345105396,
2025-07-17T08:42:58.4597207Z 			"digest": "sha256:fe14f59ec5733291d9d3c4a82afdd5f2f387bde7d9f763220a321cc899fee695"
2025-07-17T08:42:58.4597592Z 		},
2025-07-17T08:42:58.4597758Z 		{
2025-07-17T08:42:58.4598026Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4598362Z 			"size": 175,
2025-07-17T08:42:58.4598683Z 			"digest": "sha256:219c4a506b1aa90d8462496f8f39585310dac9e294ef62a311f130ec6fef10c8"
2025-07-17T08:42:58.4599056Z 		},
2025-07-17T08:42:58.4599214Z 		{
2025-07-17T08:42:58.4599479Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4599806Z 			"size": 1897,
2025-07-17T08:42:58.4600144Z 			"digest": "sha256:0a390bf78ca6647051eae3d59bef1633eef42ce7c2e97b729aa936a4f65cda7b"
2025-07-17T08:42:58.4600532Z 		},
2025-07-17T08:42:58.4600690Z 		{
2025-07-17T08:42:58.4600952Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4601292Z 			"size": 197488565,
2025-07-17T08:42:58.4601640Z 			"digest": "sha256:da75f2741b86abc1a64736fba742bed7ada087f6e14c5a74e54183533a05d4ae"
2025-07-17T08:42:58.4602028Z 		},
2025-07-17T08:42:58.4602189Z 		{
2025-07-17T08:42:58.4602457Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4602791Z 			"size": 302,
2025-07-17T08:42:58.4603127Z 			"digest": "sha256:648b993dd9d32c5275e385ad268f6cd2de85c0b6538f4806772a8bcb746c6137"
2025-07-17T08:42:58.4603512Z 		},
2025-07-17T08:42:58.4603671Z 		{
2025-07-17T08:42:58.4603944Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4604284Z 			"size": 32,
2025-07-17T08:42:58.4604619Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-07-17T08:42:58.4605140Z 		},
2025-07-17T08:42:58.4605305Z 		{
2025-07-17T08:42:58.4605576Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4605915Z 			"size": 108,
2025-07-17T08:42:58.4606262Z 			"digest": "sha256:32b8cca2c9a35e4af61d2613785bdb322625b0a3db6aaadd82816469f5958c5c"
2025-07-17T08:42:58.4606651Z 		},
2025-07-17T08:42:58.4606811Z 		{
2025-07-17T08:42:58.4607075Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-17T08:42:58.4607415Z 			"size": 54145699,
2025-07-17T08:42:58.4607759Z 			"digest": "sha256:b8c91749e9f03a297aaa508f21b46757028ef567c6ac02f4acb51a17827d9d69"
2025-07-17T08:42:58.4608141Z 		}
2025-07-17T08:42:58.4608301Z 	]
2025-07-17T08:42:58.4608462Z }
2025-07-17T08:42:58.4608637Z + exit 0
2025-07-17T08:42:58.4633384Z ##[group]Run set -eux
2025-07-17T08:42:58.4633616Z set -eux
2025-07-17T08:42:58.4634362Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2025-07-17T08:42:58.4660008Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:58.4660343Z env:
2025-07-17T08:42:58.4660544Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:58.4660918Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:42:58.4661479Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:42:58.4661979Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:42:58.4662821Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:42:58.4663574Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:42:58.4663821Z   AWS_REGION: us-east-1
2025-07-17T08:42:58.4664282Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:42:58.4664846Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:42:58.4670115Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:42:58.4670346Z ##[endgroup]
2025-07-17T08:42:58.4725323Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2025-07-17T08:42:58.4726269Z + jq --raw-output .SecretString
2025-07-17T08:42:58.4729827Z + jq -r .docker_hub_readonly_token
2025-07-17T08:42:58.4730914Z + docker login --username pytorchbot --password-stdin
2025-07-17T08:42:59.1363301Z 
2025-07-17T08:42:59.1365583Z An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::308535385114:assumed-role/gha_workflow_s3_and_ecr_read_only/GitHubActions is not authorized to perform: secretsmanager:GetSecretValue on resource: docker_hub_readonly_token because no identity-based policy allows the secretsmanager:GetSecretValue action
2025-07-17T08:42:59.2505792Z Error: Cannot perform an interactive login from a non TTY device
2025-07-17T08:42:59.2552713Z ##[error]Process completed with exit code 1.
2025-07-17T08:42:59.2667269Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2025-07-17T08:42:59.2667676Z with:
2025-07-17T08:42:59.2668280Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:59.2669020Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:59.2669348Z env:
2025-07-17T08:42:59.2669544Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:59.2669910Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:42:59.2670447Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:42:59.2670941Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:42:59.2671816Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:42:59.2672571Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:42:59.2672815Z   AWS_REGION: us-east-1
2025-07-17T08:42:59.2673142Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:42:59.2673469Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:42:59.2678743Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:42:59.2678967Z ##[endgroup]
2025-07-17T08:42:59.2693744Z ##[group]Run set -x
2025-07-17T08:42:59.2693993Z set -x
2025-07-17T08:42:59.2694186Z set +e
2025-07-17T08:42:59.2694378Z 
2025-07-17T08:42:59.2694566Z login() {
2025-07-17T08:42:59.2694982Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-07-17T08:42:59.2695435Z }
2025-07-17T08:42:59.2695616Z 
2025-07-17T08:42:59.2695802Z retry () {
2025-07-17T08:42:59.2696134Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-07-17T08:42:59.2696406Z }
2025-07-17T08:42:59.2696597Z 
2025-07-17T08:42:59.2696802Z retry login "${DOCKER_REGISTRY}"
2025-07-17T08:42:59.2697055Z 
2025-07-17T08:42:59.2697469Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024')
2025-07-17T08:42:59.2698055Z echo "Compressed size of image in MB: ${IMAGE_SIZE}"
2025-07-17T08:42:59.2698380Z 
2025-07-17T08:42:59.2698565Z set -e
2025-07-17T08:42:59.2698853Z # ignore output since only exit code is used for conditional
2025-07-17T08:42:59.2699276Z # only pull docker image if it's not available locally
2025-07-17T08:42:59.2699743Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2025-07-17T08:42:59.2700172Z   retry docker pull "${DOCKER_IMAGE}"
2025-07-17T08:42:59.2700453Z fi
2025-07-17T08:42:59.2726697Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:42:59.2727258Z env:
2025-07-17T08:42:59.2727467Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:42:59.2727836Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:42:59.2728391Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:42:59.2728895Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:42:59.2729749Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:42:59.2730502Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:42:59.2730753Z   AWS_REGION: us-east-1
2025-07-17T08:42:59.2731056Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:42:59.2731382Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:42:59.2736731Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:42:59.2737597Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:42:59.2738350Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:59.2738680Z ##[endgroup]
2025-07-17T08:42:59.2789351Z + set +e
2025-07-17T08:42:59.2789646Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:59.2790029Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:42:59.2795660Z + aws ecr get-login-password --region us-east-1
2025-07-17T08:42:59.2797462Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T08:43:00.2999321Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2025-07-17T08:43:00.3000246Z Configure a credential helper to remove this warning. See
2025-07-17T08:43:00.3001064Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2025-07-17T08:43:00.3001651Z 
2025-07-17T08:43:00.3002723Z Login Succeeded
2025-07-17T08:43:00.3051538Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:43:00.3053832Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024'
2025-07-17T08:43:01.0174864Z + IMAGE_SIZE=21749.25635433197
2025-07-17T08:43:01.0175465Z + echo 'Compressed size of image in MB: 21749.25635433197'
2025-07-17T08:43:01.0176300Z + set -e
2025-07-17T08:43:01.0176703Z Compressed size of image in MB: 21749.25635433197
2025-07-17T08:43:01.0178008Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:43:01.0504272Z Prepare all required actions
2025-07-17T08:43:01.0533536Z ##[group]Run ./.github/actions/get-workflow-job-id
2025-07-17T08:43:01.0533851Z with:
2025-07-17T08:43:01.0534327Z   github-token: ***
2025-07-17T08:43:01.0534537Z env:
2025-07-17T08:43:01.0534741Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:01.0535148Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:01.0535710Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:01.0536340Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:01.0537191Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:01.0537949Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:01.0538205Z   AWS_REGION: us-east-1
2025-07-17T08:43:01.0538512Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:01.0538848Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:01.0544136Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:01.0544361Z ##[endgroup]
2025-07-17T08:43:01.0559805Z ##[group]Run set -eux
2025-07-17T08:43:01.0560046Z set -eux
2025-07-17T08:43:01.0560432Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-07-17T08:43:01.0589740Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:43:01.0590096Z env:
2025-07-17T08:43:01.0590286Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:01.0590663Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:01.0591215Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:01.0591721Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:01.0592586Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:01.0593350Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:01.0593599Z   AWS_REGION: us-east-1
2025-07-17T08:43:01.0593892Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:01.0594252Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:01.0599576Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:01.0599906Z   GITHUB_TOKEN: ***
2025-07-17T08:43:01.0600106Z ##[endgroup]
2025-07-17T08:43:01.0647742Z + python3 .github/scripts/get_workflow_job_id.py 16337959866 gpu4f67
2025-07-17T08:43:01.5401124Z Setting output job-id=46159470976
2025-07-17T08:43:01.5401961Z Setting output job-name=rocm-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2)
2025-07-17T08:43:01.5595753Z Prepare all required actions
2025-07-17T08:43:01.5596167Z Getting action download info
2025-07-17T08:43:01.7200595Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2025-07-17T08:43:02.3599403Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093)
2025-07-17T08:43:02.9842463Z ##[group]Run ./.github/actions/download-build-artifacts
2025-07-17T08:43:02.9842786Z with:
2025-07-17T08:43:02.9842998Z   name: linux-jammy-rocm-py3.10
2025-07-17T08:43:02.9843283Z   s3-bucket: gha-artifacts
2025-07-17T08:43:02.9843513Z env:
2025-07-17T08:43:02.9843696Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:02.9844066Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:02.9844616Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:02.9845144Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:02.9845997Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:02.9846753Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:02.9846996Z   AWS_REGION: us-east-1
2025-07-17T08:43:02.9847318Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:02.9847665Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:02.9852982Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:02.9853215Z ##[endgroup]
2025-07-17T08:43:02.9937335Z ##[group]Run seemethere/download-artifact-s3@v4
2025-07-17T08:43:02.9937734Z with:
2025-07-17T08:43:02.9937980Z   name: linux-jammy-rocm-py3.10
2025-07-17T08:43:02.9938293Z   s3-bucket: gha-artifacts
2025-07-17T08:43:02.9938567Z   region: us-east-1
2025-07-17T08:43:02.9938795Z env:
2025-07-17T08:43:02.9939011Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:02.9939450Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:02.9940102Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:02.9940712Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:02.9941756Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:02.9942612Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:02.9942847Z   AWS_REGION: us-east-1
2025-07-17T08:43:02.9943410Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:02.9943741Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:02.9948975Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:02.9949184Z ##[endgroup]
2025-07-17T08:43:03.4908181Z (node:3990734) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-07-17T08:43:03.4908745Z 
2025-07-17T08:43:03.4909007Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-07-17T08:43:03.4909613Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-07-17T08:43:03.4910241Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-07-17T08:43:03.6366447Z Found 1 objects with prefix pytorch/pytorch/16337959866/linux-jammy-rocm-py3.10/
2025-07-17T08:43:03.6367534Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-07-17T08:43:34.6653082Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-07-17T08:43:34.6661673Z Artifact download has finished successfully
2025-07-17T08:43:34.7059329Z ##[group]Run unzip -o artifacts.zip
2025-07-17T08:43:34.7059641Z unzip -o artifacts.zip
2025-07-17T08:43:34.7086382Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:43:34.7086730Z env:
2025-07-17T08:43:34.7086924Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:34.7087567Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:34.7088139Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:34.7088648Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:34.7089489Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:34.7090238Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:34.7090490Z   AWS_REGION: us-east-1
2025-07-17T08:43:34.7090801Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:34.7091120Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:34.7096473Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:34.7096704Z ##[endgroup]
2025-07-17T08:43:34.7166628Z Archive:  artifacts.zip
2025-07-17T08:43:34.7167333Z    creating: dist/
2025-07-17T08:43:40.9163165Z   inflating: dist/torch-2.9.0a0+gita38f433-cp310-cp310-linux_x86_64.whl  
2025-07-17T08:43:40.9313895Z   inflating: dist/.ninja_log         
2025-07-17T08:43:40.9314580Z    creating: build/custom_test_artifacts/
2025-07-17T08:43:40.9316050Z    creating: build/custom_test_artifacts/custom-op-build/
2025-07-17T08:43:40.9316883Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2025-07-17T08:43:40.9317771Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2025-07-17T08:43:40.9320965Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-07-17T08:43:40.9321606Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/
2025-07-17T08:43:40.9322158Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-07-17T08:43:40.9322761Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/
2025-07-17T08:43:40.9323352Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-07-17T08:43:40.9325244Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-07-17T08:43:40.9326956Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-07-17T08:43:40.9327855Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-07-17T08:43:40.9328458Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-07-17T08:43:40.9329040Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-07-17T08:43:40.9331472Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-07-17T08:43:40.9332915Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-07-17T08:43:40.9333843Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-07-17T08:43:40.9335494Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-07-17T08:43:40.9337176Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-07-17T08:43:40.9337827Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2025-07-17T08:43:40.9338339Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2025-07-17T08:43:40.9338877Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2025-07-17T08:43:40.9339430Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2025-07-17T08:43:40.9340042Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2025-07-17T08:43:40.9340741Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2025-07-17T08:43:40.9341689Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2025-07-17T08:43:40.9342338Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2025-07-17T08:43:40.9342982Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2025-07-17T08:43:40.9343628Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2025-07-17T08:43:40.9344295Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2025-07-17T08:43:40.9344943Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2025-07-17T08:43:40.9345599Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2025-07-17T08:43:40.9367963Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2025-07-17T08:43:40.9578081Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2025-07-17T08:43:40.9579119Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2025-07-17T08:43:40.9580233Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2025-07-17T08:43:40.9581634Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2025-07-17T08:43:40.9583041Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2025-07-17T08:43:40.9584369Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2025-07-17T08:43:40.9585635Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2025-07-17T08:43:40.9586788Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2025-07-17T08:43:40.9587945Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2025-07-17T08:43:40.9589081Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2025-07-17T08:43:40.9590196Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2025-07-17T08:43:40.9607254Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2025-07-17T08:43:40.9695123Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2025-07-17T08:43:40.9696132Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-07-17T08:43:40.9696789Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2025-07-17T08:43:40.9697387Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2025-07-17T08:43:40.9697944Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2025-07-17T08:43:40.9699355Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2025-07-17T08:43:40.9699936Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/InstallScripts.json  
2025-07-17T08:43:40.9700534Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_outer_vec.cc  
2025-07-17T08:43:40.9701099Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_vec_ext.cc  
2025-07-17T08:43:40.9702770Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2025-07-17T08:43:40.9703595Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2025-07-17T08:43:40.9704215Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2025-07-17T08:43:40.9879649Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2025-07-17T08:43:40.9939991Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2025-07-17T08:43:40.9940775Z    creating: build/custom_test_artifacts/jit-hook-build/
2025-07-17T08:43:40.9941460Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2025-07-17T08:43:40.9942274Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2025-07-17T08:43:40.9945677Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-07-17T08:43:40.9946623Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/
2025-07-17T08:43:40.9947522Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-07-17T08:43:40.9948509Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/
2025-07-17T08:43:40.9949462Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-07-17T08:43:40.9950573Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-07-17T08:43:40.9951751Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-07-17T08:43:40.9952987Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-07-17T08:43:40.9954175Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-07-17T08:43:40.9955237Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-07-17T08:43:40.9955978Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-07-17T08:43:40.9956711Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-07-17T08:43:40.9957706Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-07-17T08:43:40.9959360Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-07-17T08:43:40.9960840Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-07-17T08:43:40.9961483Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2025-07-17T08:43:40.9961988Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2025-07-17T08:43:40.9962524Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2025-07-17T08:43:40.9963079Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2025-07-17T08:43:40.9963911Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2025-07-17T08:43:40.9964630Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2025-07-17T08:43:40.9965331Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2025-07-17T08:43:40.9965975Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2025-07-17T08:43:40.9966632Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2025-07-17T08:43:40.9967322Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2025-07-17T08:43:40.9967999Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2025-07-17T08:43:40.9968665Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2025-07-17T08:43:40.9969329Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2025-07-17T08:43:40.9991465Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2025-07-17T08:43:41.0061320Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2025-07-17T08:43:41.0062576Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-07-17T08:43:41.0063666Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2025-07-17T08:43:41.0064663Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2025-07-17T08:43:41.0065588Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2025-07-17T08:43:41.0066467Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2025-07-17T08:43:41.0067440Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/InstallScripts.json  
2025-07-17T08:43:41.0068416Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_outer_vec.cc  
2025-07-17T08:43:41.0069313Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_vec_ext.cc  
2025-07-17T08:43:41.0070177Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2025-07-17T08:43:41.0070960Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2025-07-17T08:43:41.0071764Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2025-07-17T08:43:41.0111021Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2025-07-17T08:43:41.0111812Z    creating: build/custom_test_artifacts/custom-backend-build/
2025-07-17T08:43:41.0112590Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2025-07-17T08:43:41.0113492Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2025-07-17T08:43:41.0116913Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-07-17T08:43:41.0117940Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/
2025-07-17T08:43:41.0118952Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-07-17T08:43:41.0120019Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/
2025-07-17T08:43:41.0121068Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-07-17T08:43:41.0122260Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-07-17T08:43:41.0123101Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-07-17T08:43:41.0123765Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-07-17T08:43:41.0124625Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-07-17T08:43:41.0125253Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-07-17T08:43:41.0126608Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-07-17T08:43:41.0128127Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-07-17T08:43:41.0128990Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-07-17T08:43:41.0130668Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-07-17T08:43:41.0132064Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-07-17T08:43:41.0132769Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2025-07-17T08:43:41.0133323Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2025-07-17T08:43:41.0133904Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2025-07-17T08:43:41.0134709Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2025-07-17T08:43:41.0135427Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2025-07-17T08:43:41.0136289Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2025-07-17T08:43:41.0137048Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2025-07-17T08:43:41.0137744Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2025-07-17T08:43:41.0138463Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2025-07-17T08:43:41.0139185Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2025-07-17T08:43:41.0139909Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2025-07-17T08:43:41.0140630Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2025-07-17T08:43:41.0141357Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2025-07-17T08:43:41.0144138Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2025-07-17T08:43:41.0275660Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2025-07-17T08:43:41.0276887Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2025-07-17T08:43:41.0278116Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2025-07-17T08:43:41.0279498Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2025-07-17T08:43:41.0280863Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2025-07-17T08:43:41.0282112Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2025-07-17T08:43:41.0283165Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2025-07-17T08:43:41.0283916Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2025-07-17T08:43:41.0284671Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2025-07-17T08:43:41.0285635Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2025-07-17T08:43:41.0286380Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2025-07-17T08:43:41.0305153Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2025-07-17T08:43:41.0364408Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2025-07-17T08:43:41.0365802Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-07-17T08:43:41.0367041Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2025-07-17T08:43:41.0368118Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2025-07-17T08:43:41.0369119Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2025-07-17T08:43:41.0370081Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2025-07-17T08:43:41.0371121Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/InstallScripts.json  
2025-07-17T08:43:41.0372458Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_outer_vec.cc  
2025-07-17T08:43:41.0373300Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_vec_ext.cc  
2025-07-17T08:43:41.0373851Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2025-07-17T08:43:41.0374350Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2025-07-17T08:43:41.0374863Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2025-07-17T08:43:41.0482407Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2025-07-17T08:43:41.0524590Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2025-07-17T08:43:41.0525295Z    creating: build/lib/
2025-07-17T08:43:41.0613834Z   inflating: build/lib/libprotobuf-lite.a  
2025-07-17T08:43:41.1090152Z   inflating: build/lib/libprotobuf.a  
2025-07-17T08:43:41.1621720Z   inflating: build/lib/libprotoc.a   
2025-07-17T08:43:41.1632180Z   inflating: build/lib/libpthreadpool.a  
2025-07-17T08:43:41.1640719Z   inflating: build/lib/libcpuinfo.a  
2025-07-17T08:43:41.1649068Z   inflating: build/lib/libcpuinfo_internals.a  
2025-07-17T08:43:41.1649972Z   inflating: build/lib/libclog.a     
2025-07-17T08:43:41.1670395Z   inflating: build/lib/libpytorch_qnnpack.a  
2025-07-17T08:43:41.1672616Z   inflating: build/lib/libnnpack_reference_layers.a  
2025-07-17T08:43:41.1691535Z   inflating: build/lib/libnnpack.a   
2025-07-17T08:43:41.1890460Z   inflating: build/lib/libmicrokernels-prod.a  
2025-07-17T08:43:41.2827066Z   inflating: build/lib/libmicrokernels-all.a  
2025-07-17T08:43:41.2902334Z   inflating: build/lib/libgtest.a    
2025-07-17T08:43:41.2920679Z   inflating: build/lib/libgmock.a    
2025-07-17T08:43:41.2921282Z   inflating: build/lib/libgtest_main.a  
2025-07-17T08:43:41.2922322Z   inflating: build/lib/libgmock_main.a  
2025-07-17T08:43:41.3019076Z   inflating: build/lib/libXNNPACK.a  
2025-07-17T08:43:41.3099544Z   inflating: build/lib/libbenchmark.a  
2025-07-17T08:43:41.3100443Z   inflating: build/lib/libbenchmark_main.a  
2025-07-17T08:43:41.3101166Z   inflating: build/lib/libjitprofiling.a  
2025-07-17T08:43:41.3109765Z   inflating: build/lib/libittnotify.a  
2025-07-17T08:43:41.3178368Z   inflating: build/lib/libasmjit.a   
2025-07-17T08:43:41.4475882Z   inflating: build/lib/libfbgemm.a   
2025-07-17T08:43:41.4504733Z   inflating: build/lib/libtensorpipe_uv.a  
2025-07-17T08:43:41.5103028Z   inflating: build/lib/libtensorpipe.a  
2025-07-17T08:43:41.5232030Z   inflating: build/lib/libgloo.a     
2025-07-17T08:43:41.5283102Z   inflating: build/lib/libonnx_proto.a  
2025-07-17T08:43:41.5754822Z   inflating: build/lib/libgloo_hip.a  
2025-07-17T08:43:41.6529468Z   inflating: build/lib/libonnx.a     
2025-07-17T08:43:42.7658786Z   inflating: build/lib/libdnnl.a     
2025-07-17T08:43:42.7678470Z   inflating: build/lib/libfmt.a      
2025-07-17T08:43:42.7988180Z   inflating: build/lib/libkineto.a   
2025-07-17T08:43:42.8100814Z   inflating: build/lib/libc10.so     
2025-07-17T08:43:42.8102099Z   inflating: build/lib/libtorch_global_deps.so  
2025-07-17T08:43:42.8156670Z   inflating: build/lib/libc10_hip.so  
2025-07-17T08:43:42.8158344Z   inflating: build/lib/libcaffe2_nvrtc.so  
2025-07-17T08:43:46.3956938Z   inflating: build/lib/libtorch_cpu.so  
2025-07-17T08:43:46.3960930Z   inflating: build/lib/libshm.so     
2025-07-17T08:43:47.3046131Z   inflating: build/lib/libtorch_hip.so  
2025-07-17T08:43:47.3047310Z   inflating: build/lib/libtorch.so   
2025-07-17T08:43:47.3066929Z   inflating: build/lib/libjitbackend_test.so  
2025-07-17T08:43:47.3092539Z   inflating: build/lib/libbackend_with_compiler.so  
2025-07-17T08:43:47.3167255Z   inflating: build/lib/libtorchbind_test.so  
2025-07-17T08:43:47.3195615Z   inflating: build/lib/libaoti_custom_ops.so  
2025-07-17T08:43:47.5372431Z   inflating: build/lib/libtorch_python.so  
2025-07-17T08:43:47.5407535Z   inflating: build/lib/libnnapi_backend.so  
2025-07-17T08:43:47.5408145Z    creating: build/bin/
2025-07-17T08:43:47.5409244Z    creating: build/bin/CMakeFiles/
2025-07-17T08:43:47.5409860Z   inflating: build/bin/cmake_install.cmake  
2025-07-17T08:43:47.5410391Z   inflating: build/bin/CTestTestfile.cmake  
2025-07-17T08:43:47.5890734Z   inflating: build/bin/protoc-3.13.0.0  
2025-07-17T08:43:47.6372606Z   inflating: build/bin/protoc        
2025-07-17T08:43:47.6430960Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2025-07-17T08:43:47.6490954Z   inflating: build/bin/c10_DeviceGuard_test  
2025-07-17T08:43:47.6551469Z   inflating: build/bin/c10_Device_test  
2025-07-17T08:43:47.6620213Z   inflating: build/bin/c10_DispatchKeySet_test  
2025-07-17T08:43:47.6682939Z   inflating: build/bin/c10_Scalar_test  
2025-07-17T08:43:47.6740517Z   inflating: build/bin/c10_StreamGuard_test  
2025-07-17T08:43:47.6800068Z   inflating: build/bin/c10_SymInt_test  
2025-07-17T08:43:47.6863405Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2025-07-17T08:43:47.6920720Z   inflating: build/bin/c10_ConstexprCrc_test  
2025-07-17T08:43:47.6985518Z   inflating: build/bin/c10_InlineStreamGuard_test  
2025-07-17T08:43:47.7043564Z   inflating: build/bin/c10_ArrayRef_test  
2025-07-17T08:43:47.7108826Z   inflating: build/bin/c10_SizesAndStrides_test  
2025-07-17T08:43:47.7170507Z   inflating: build/bin/c10_Bitset_test  
2025-07-17T08:43:47.7251277Z   inflating: build/bin/c10_cow_test  
2025-07-17T08:43:47.7317612Z   inflating: build/bin/c10_Enumerate_test  
2025-07-17T08:43:47.7376445Z   inflating: build/bin/c10_DeadlockDetection_test  
2025-07-17T08:43:47.7435658Z   inflating: build/bin/c10_Half_test  
2025-07-17T08:43:47.7497315Z   inflating: build/bin/c10_IntrusiveList_test  
2025-07-17T08:43:47.7562279Z   inflating: build/bin/c10_LeftRight_test  
2025-07-17T08:43:47.7627059Z   inflating: build/bin/c10_Metaprogramming_test  
2025-07-17T08:43:47.7689141Z   inflating: build/bin/c10_NetworkFlow_test  
2025-07-17T08:43:47.7747262Z   inflating: build/bin/c10_Semaphore_test  
2025-07-17T08:43:47.7811805Z   inflating: build/bin/c10_ThreadLocal_test  
2025-07-17T08:43:47.7872453Z   inflating: build/bin/c10_TypeIndex_test  
2025-07-17T08:43:47.7930890Z   inflating: build/bin/c10_Synchronized_test  
2025-07-17T08:43:47.7990520Z   inflating: build/bin/c10_TypeList_test  
2025-07-17T08:43:47.8047920Z   inflating: build/bin/c10_TypeTraits_test  
2025-07-17T08:43:47.8108281Z   inflating: build/bin/c10_accumulate_test  
2025-07-17T08:43:47.8174583Z   inflating: build/bin/c10_complex_math_test  
2025-07-17T08:43:47.8233862Z   inflating: build/bin/c10_bit_cast_test  
2025-07-17T08:43:47.8298533Z   inflating: build/bin/c10_bfloat16_test  
2025-07-17T08:43:47.8356373Z   inflating: build/bin/c10_error_test  
2025-07-17T08:43:47.8420464Z   inflating: build/bin/c10_complex_test  
2025-07-17T08:43:47.8481563Z   inflating: build/bin/c10_exception_test  
2025-07-17T08:43:47.8540469Z   inflating: build/bin/c10_flags_test  
2025-07-17T08:43:47.8599215Z   inflating: build/bin/c10_generic_math_test  
2025-07-17T08:43:47.8658953Z   inflating: build/bin/c10_irange_test  
2025-07-17T08:43:47.8845855Z   inflating: build/bin/c10_intrusive_ptr_test  
2025-07-17T08:43:47.8908205Z   inflating: build/bin/c10_lazy_test  
2025-07-17T08:43:47.8974438Z   inflating: build/bin/c10_logging_test  
2025-07-17T08:43:47.9060916Z   inflating: build/bin/c10_optional_test  
2025-07-17T08:43:47.9122915Z   inflating: build/bin/c10_registry_test  
2025-07-17T08:43:47.9194497Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2025-07-17T08:43:47.9370173Z   inflating: build/bin/c10_small_vector_test  
2025-07-17T08:43:47.9430343Z   inflating: build/bin/c10_ssize_test  
2025-07-17T08:43:47.9496178Z   inflating: build/bin/c10_string_util_test  
2025-07-17T08:43:47.9553528Z   inflating: build/bin/c10_string_view_test  
2025-07-17T08:43:47.9612090Z   inflating: build/bin/c10_tempfile_test  
2025-07-17T08:43:47.9663011Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2025-07-17T08:43:47.9728350Z   inflating: build/bin/c10_typeid_test  
2025-07-17T08:43:47.9786228Z   inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test  
2025-07-17T08:43:47.9843268Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream  
2025-07-17T08:43:47.9900720Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device  
2025-07-17T08:43:47.9957893Z   inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes  
2025-07-17T08:43:48.0015371Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads  
2025-07-17T08:43:48.0072581Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks  
2025-07-17T08:43:48.0130076Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block  
2025-07-17T08:43:48.0188015Z   inflating: build/bin/c10_hip_HIPTest  
2025-07-17T08:43:48.1405101Z   inflating: build/bin/vec_test_all_types_AVX512  
2025-07-17T08:43:48.2606606Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2025-07-17T08:43:48.3829987Z   inflating: build/bin/vec_test_all_types_AVX2  
2025-07-17T08:43:48.3891323Z   inflating: build/bin/BackoffTest   
2025-07-17T08:43:48.3953294Z   inflating: build/bin/FileStoreTest  
2025-07-17T08:43:48.4019394Z   inflating: build/bin/TCPStoreTest  
2025-07-17T08:43:48.4082164Z   inflating: build/bin/HashStoreTest  
2025-07-17T08:43:48.4158317Z   inflating: build/bin/ProcessGroupGlooTest  
2025-07-17T08:43:48.4161375Z   inflating: build/bin/example_allreduce  
2025-07-17T08:43:48.4165455Z   inflating: build/bin/torch_shm_manager  
2025-07-17T08:43:48.4227919Z   inflating: build/bin/static_runtime_bench  
2025-07-17T08:43:48.4312825Z   inflating: build/bin/Dict_test     
2025-07-17T08:43:48.4373980Z   inflating: build/bin/Dimname_test  
2025-07-17T08:43:48.4449627Z   inflating: build/bin/MaybeOwned_test  
2025-07-17T08:43:48.4515881Z   inflating: build/bin/NamedTensor_test  
2025-07-17T08:43:48.4583862Z   inflating: build/bin/apply_utils_test  
2025-07-17T08:43:48.4652203Z   inflating: build/bin/atest         
2025-07-17T08:43:48.4716517Z   inflating: build/bin/broadcast_test  
2025-07-17T08:43:48.4776255Z   inflating: build/bin/cpu_allocator_test  
2025-07-17T08:43:48.4848984Z   inflating: build/bin/basic         
2025-07-17T08:43:48.4916529Z   inflating: build/bin/cpu_generator_test  
2025-07-17T08:43:48.4978489Z   inflating: build/bin/cpu_profiling_allocator_test  
2025-07-17T08:43:48.5259369Z   inflating: build/bin/static_runtime_test  
2025-07-17T08:43:48.5319005Z   inflating: build/bin/dlconvertor_test  
2025-07-17T08:43:48.5385401Z   inflating: build/bin/extension_backend_test  
2025-07-17T08:43:48.5449960Z   inflating: build/bin/half_test     
2025-07-17T08:43:48.5508407Z   inflating: build/bin/lazy_tensor_test  
2025-07-17T08:43:48.5570745Z   inflating: build/bin/memory_format_test  
2025-07-17T08:43:48.5633149Z   inflating: build/bin/math_kernel_test  
2025-07-17T08:43:48.5695595Z   inflating: build/bin/memory_overlapping_test  
2025-07-17T08:43:48.5755247Z   inflating: build/bin/operator_name_test  
2025-07-17T08:43:48.5814548Z   inflating: build/bin/operators_test  
2025-07-17T08:43:48.5923279Z   inflating: build/bin/ivalue_test   
2025-07-17T08:43:48.5984039Z   inflating: build/bin/packedtensoraccessor_test  
2025-07-17T08:43:48.6049028Z   inflating: build/bin/native_test   
2025-07-17T08:43:48.6153974Z   inflating: build/bin/cpu_rng_test  
2025-07-17T08:43:48.6216031Z   inflating: build/bin/mobile_memory_cleanup  
2025-07-17T08:43:48.6282700Z   inflating: build/bin/quantized_test  
2025-07-17T08:43:48.6342307Z   inflating: build/bin/reportMemoryUsage_test  
2025-07-17T08:43:48.6419627Z   inflating: build/bin/pow_test      
2025-07-17T08:43:48.6478132Z   inflating: build/bin/reduce_ops_test  
2025-07-17T08:43:48.6544160Z   inflating: build/bin/scalar_tensor_test  
2025-07-17T08:43:48.6604018Z   inflating: build/bin/StorageUtils_test  
2025-07-17T08:43:48.6672018Z   inflating: build/bin/scalar_test   
2025-07-17T08:43:48.6736355Z   inflating: build/bin/type_ptr_test  
2025-07-17T08:43:48.6797197Z   inflating: build/bin/stride_properties_test  
2025-07-17T08:43:48.6800329Z   inflating: build/bin/thread_init_test  
2025-07-17T08:43:48.6861324Z   inflating: build/bin/undefined_tensor_test  
2025-07-17T08:43:48.6924572Z   inflating: build/bin/test_parallel  
2025-07-17T08:43:48.6926069Z   inflating: build/bin/verify_api_visibility  
2025-07-17T08:43:48.7017555Z   inflating: build/bin/tensor_iterator_test  
2025-07-17T08:43:48.7086551Z   inflating: build/bin/IListRef_test  
2025-07-17T08:43:48.7207890Z   inflating: build/bin/List_test     
2025-07-17T08:43:48.7267867Z   inflating: build/bin/weakref_test  
2025-07-17T08:43:48.7328160Z   inflating: build/bin/xla_tensor_test  
2025-07-17T08:43:48.7388255Z   inflating: build/bin/wrapdim_test  
2025-07-17T08:43:48.7496557Z   inflating: build/bin/kernel_function_test  
2025-07-17T08:43:48.7632020Z   inflating: build/bin/kernel_function_legacy_test  
2025-07-17T08:43:48.7773966Z   inflating: build/bin/kernel_lambda_legacy_test  
2025-07-17T08:43:48.7843843Z   inflating: build/bin/kernel_stackbased_test  
2025-07-17T08:43:48.7960192Z   inflating: build/bin/kernel_lambda_test  
2025-07-17T08:43:48.8068431Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2025-07-17T08:43:48.8128177Z   inflating: build/bin/CppSignature_test  
2025-07-17T08:43:48.8185687Z   inflating: build/bin/op_allowlist_test  
2025-07-17T08:43:48.8254807Z   inflating: build/bin/type_test     
2025-07-17T08:43:48.8602773Z   inflating: build/bin/op_registration_test  
2025-07-17T08:43:48.8683016Z   inflating: build/bin/legacy_vmap_test  
2025-07-17T08:43:48.8760135Z   inflating: build/bin/inline_container_test  
2025-07-17T08:43:48.8817568Z   inflating: build/bin/hip_complex_math_test  
2025-07-17T08:43:48.8879221Z   inflating: build/bin/hip_apply_test  
2025-07-17T08:43:48.8936624Z   inflating: build/bin/hip_complex_test  
2025-07-17T08:43:48.8994032Z   inflating: build/bin/hip_distributions_test  
2025-07-17T08:43:48.9051411Z   inflating: build/bin/hip_generator_test  
2025-07-17T08:43:48.9108705Z   inflating: build/bin/hip_half_test  
2025-07-17T08:43:48.9166101Z   inflating: build/bin/hip_integer_divider_test  
2025-07-17T08:43:48.9224038Z   inflating: build/bin/hip_optional_test  
2025-07-17T08:43:48.9282088Z   inflating: build/bin/hip_packedtensoraccessor_test  
2025-07-17T08:43:48.9339328Z   inflating: build/bin/hip_vectorized_test  
2025-07-17T08:43:48.9415520Z   inflating: build/bin/KernelFunction_test  
2025-07-17T08:43:48.9479592Z   inflating: build/bin/backend_fallback_test  
2025-07-17T08:43:48.9539927Z   inflating: build/bin/hip_dlconvertor_test  
2025-07-17T08:43:49.0734620Z   inflating: build/bin/test_jit      
2025-07-17T08:43:49.1041675Z   inflating: build/bin/test_nativert  
2025-07-17T08:43:49.1055196Z   inflating: build/bin/tutorial_tensorexpr  
2025-07-17T08:43:49.1119334Z   inflating: build/bin/test_dist_autograd  
2025-07-17T08:43:49.1197149Z   inflating: build/bin/test_cpp_rpc  
2025-07-17T08:43:49.2054910Z   inflating: build/bin/test_tensorexpr  
2025-07-17T08:43:49.2057661Z   inflating: build/bin/parallel_benchmark  
2025-07-17T08:43:49.2134008Z   inflating: build/bin/test_mobile_nnc  
2025-07-17T08:43:49.3392386Z   inflating: build/bin/test_api      
2025-07-17T08:43:49.3401729Z   inflating: build/bin/aot_model_compiler_test  
2025-07-17T08:43:49.3786152Z   inflating: build/bin/test_lazy     
2025-07-17T08:43:49.3786647Z    creating: .additional_ci_files/
2025-07-17T08:43:49.3958635Z   inflating: .additional_ci_files/test-times.json  
2025-07-17T08:43:49.4616157Z   inflating: .additional_ci_files/test-class-times.json  
2025-07-17T08:43:49.4657838Z ##[group]Run rm artifacts.zip
2025-07-17T08:43:49.4658115Z rm artifacts.zip
2025-07-17T08:43:49.4684358Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:43:49.4684701Z env:
2025-07-17T08:43:49.4684899Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:49.4685275Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:49.4685834Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:49.4686584Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:49.4687428Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:49.4688168Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:49.4688416Z   AWS_REGION: us-east-1
2025-07-17T08:43:49.4688728Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:49.4689118Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:49.4694378Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:49.4694615Z ##[endgroup]
2025-07-17T08:43:49.8127865Z ##[group]Run df -H
2025-07-17T08:43:49.8128105Z df -H
2025-07-17T08:43:49.8157145Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:43:49.8157487Z env:
2025-07-17T08:43:49.8157693Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:49.8158065Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:49.8158645Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:49.8159153Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:49.8159998Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:49.8160754Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:49.8161005Z   AWS_REGION: us-east-1
2025-07-17T08:43:49.8161338Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:49.8161663Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:49.8166928Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:49.8167155Z ##[endgroup]
2025-07-17T08:43:49.8280367Z Filesystem                                       Size  Used Avail Use% Mounted on
2025-07-17T08:43:49.8281169Z tmpfs                                            109G   33M  109G   1% /run
2025-07-17T08:43:49.8281869Z /dev/nvme0n1p2                                   944G   81G  816G   9% /
2025-07-17T08:43:49.8284677Z tmpfs                                            542G   33k  542G   1% /dev/shm
2025-07-17T08:43:49.8285141Z tmpfs                                            5.3M     0  5.3M   0% /run/lock
2025-07-17T08:43:49.8285616Z /dev/nvme0n1p1                                   536M  6.4M  530M   2% /boot/efi
2025-07-17T08:43:49.8286012Z /dev/nvme1n1p1                                   3.8T  607G  3.0T  17% /media/4TB
2025-07-17T08:43:49.8286408Z tmpfs                                            109G   33k  109G   1% /run/user/1307800118
2025-07-17T08:43:49.8287074Z 172.18.148.8:/export/amd2                        5.5T  278G  5.3T   6% /mnt
2025-07-17T08:43:49.8287569Z pure1.jax.cs.cpe.ice.amd.com:/homes/amd-pytorch  108G  327M  108G   1% /home/amd-pytorch
2025-07-17T08:43:49.8288067Z 172.18.148.15:/GroupStorage                      132T  132T  788G 100% /groups
2025-07-17T08:43:49.8288530Z 172.18.148.15:/GroupStorage/Scratch              2.6T  1.8T  788G  70% /scratch
2025-07-17T08:43:49.8289029Z pure1.jax.cs.cpe.ice.amd.com:/homes/nlingamp     108G  2.2G  106G   3% /home/nlingamp
2025-07-17T08:43:49.8289536Z pure1.jax.cs.cpe.ice.amd.com:/homes/dhonnapp     108G   13G   95G  12% /home/dhonnapp
2025-07-17T08:43:49.8290011Z pure1.jax.cs.cpe.ice.amd.com:/homes/jlay         108G   19G   89G  18% /home/jlay
2025-07-17T08:43:49.8320669Z Prepare all required actions
2025-07-17T08:43:49.8321065Z Getting action download info
2025-07-17T08:43:49.9847829Z ##[group]Run ./.github/actions/download-td-artifacts
2025-07-17T08:43:49.9848136Z with:
2025-07-17T08:43:49.9848314Z env:
2025-07-17T08:43:49.9848491Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:49.9848857Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:49.9849390Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:49.9849878Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:49.9850703Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:49.9851467Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:49.9851709Z   AWS_REGION: us-east-1
2025-07-17T08:43:49.9852005Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:49.9852331Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:49.9857674Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:49.9857893Z ##[endgroup]
2025-07-17T08:43:49.9884599Z ##[group]Run seemethere/download-artifact-s3@v4
2025-07-17T08:43:49.9884910Z with:
2025-07-17T08:43:49.9885095Z   name: td_results
2025-07-17T08:43:49.9885318Z   s3-bucket: gha-artifacts
2025-07-17T08:43:49.9885550Z   region: us-east-1
2025-07-17T08:43:49.9885747Z env:
2025-07-17T08:43:49.9885938Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:49.9886303Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:49.9886852Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:49.9887358Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:49.9888201Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:49.9888951Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:49.9889208Z   AWS_REGION: us-east-1
2025-07-17T08:43:49.9889488Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:49.9889822Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:49.9895086Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:49.9895325Z ##[endgroup]
2025-07-17T08:43:50.4764262Z (node:3990789) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-07-17T08:43:50.4764976Z 
2025-07-17T08:43:50.4765270Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-07-17T08:43:50.4766050Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-07-17T08:43:50.4766878Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-07-17T08:43:50.6427037Z Found 0 objects with prefix pytorch/pytorch/16337959866/td_results/
2025-07-17T08:43:50.6433784Z Artifact download has finished successfully
2025-07-17T08:43:50.6795156Z ##[group]Run mkdir -p .additional_ci_files
2025-07-17T08:43:50.6795506Z mkdir -p .additional_ci_files
2025-07-17T08:43:50.6795886Z mv td_results.json .additional_ci_files/td_results.json || true
2025-07-17T08:43:50.6824021Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:43:50.6824365Z env:
2025-07-17T08:43:50.6824570Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:50.6824947Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:50.6825520Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:50.6826036Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:50.6826896Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:50.6827926Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:50.6828191Z   AWS_REGION: us-east-1
2025-07-17T08:43:50.6828504Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:50.6828831Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:50.6834119Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:50.6834351Z ##[endgroup]
2025-07-17T08:43:50.6929690Z mv: cannot stat 'td_results.json': No such file or directory
2025-07-17T08:43:50.6969625Z ##[group]Run .github/scripts/parse_ref.py
2025-07-17T08:43:50.6969993Z .github/scripts/parse_ref.py
2025-07-17T08:43:50.6999836Z shell: /usr/bin/bash -e {0}
2025-07-17T08:43:50.7000093Z env:
2025-07-17T08:43:50.7000302Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:50.7000698Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:50.7001263Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:50.7001778Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:50.7002631Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:50.7003410Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:50.7003667Z   AWS_REGION: us-east-1
2025-07-17T08:43:50.7003966Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:50.7004335Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:50.7009620Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:50.7009850Z ##[endgroup]
2025-07-17T08:43:50.7248887Z Setting output branch=main
2025-07-17T08:43:50.7369700Z Prepare all required actions
2025-07-17T08:43:50.7370155Z Getting action download info
2025-07-17T08:43:50.8915722Z ##[group]Run ./.github/actions/filter-test-configs
2025-07-17T08:43:50.8916040Z with:
2025-07-17T08:43:50.8916465Z   github-token: ***
2025-07-17T08:43:50.8917075Z   test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}]}
2025-07-17T08:43:50.8917841Z   job-name: rocm-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2)
2025-07-17T08:43:50.8918213Z env:
2025-07-17T08:43:50.8918447Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:50.8918825Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:50.8919374Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:50.8919882Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:50.8920757Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:50.8921520Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:50.8921772Z   AWS_REGION: us-east-1
2025-07-17T08:43:50.8922046Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:50.8922399Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:50.8927642Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:50.8927864Z ##[endgroup]
2025-07-17T08:43:50.8980807Z ##[group]Run nick-fields/retry@v3.0.0
2025-07-17T08:43:50.8981079Z with:
2025-07-17T08:43:50.8981542Z   shell: bash
2025-07-17T08:43:50.8981763Z   timeout_minutes: 10
2025-07-17T08:43:50.8981996Z   max_attempts: 5
2025-07-17T08:43:50.8982218Z   retry_wait_seconds: 30
2025-07-17T08:43:50.8982916Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2025-07-17T08:43:50.8983665Z   polling_interval_seconds: 1
2025-07-17T08:43:50.8983925Z   warning_on_retry: true
2025-07-17T08:43:50.8984164Z   continue_on_error: false
2025-07-17T08:43:50.8984400Z env:
2025-07-17T08:43:50.8984593Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:50.8984966Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:50.8985506Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:50.8986021Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:50.8986880Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:50.8987645Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:50.8987923Z   AWS_REGION: us-east-1
2025-07-17T08:43:50.8988225Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:50.8988567Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:50.8993904Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:50.8994260Z   GITHUB_TOKEN: ***
2025-07-17T08:43:50.8994488Z ##[endgroup]
2025-07-17T08:43:50.9818034Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2025-07-17T08:43:51.2826473Z Defaulting to user installation because normal site-packages is not writeable
2025-07-17T08:43:51.3940077Z Requirement already satisfied: requests==2.27.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (2.27.1)
2025-07-17T08:43:51.3944895Z Requirement already satisfied: pyyaml==6.0.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (6.0.1)
2025-07-17T08:43:51.4044890Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20)
2025-07-17T08:43:51.4050363Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5)
2025-07-17T08:43:51.4059600Z Requirement already satisfied: charset-normalizer~=2.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12)
2025-07-17T08:43:51.4069958Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3)
2025-07-17T08:43:51.9822837Z Command completed after 1 attempt(s).
2025-07-17T08:43:51.9916732Z ##[group]Run set -x
2025-07-17T08:43:51.9916986Z set -x
2025-07-17T08:43:51.9917187Z 
2025-07-17T08:43:51.9917538Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-07-17T08:43:51.9917977Z # in runner workspace
2025-07-17T08:43:51.9918357Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2025-07-17T08:43:51.9942945Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:43:51.9943304Z env:
2025-07-17T08:43:51.9943505Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:51.9943925Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:51.9944488Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:51.9945023Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:51.9945887Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:51.9946646Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:51.9946909Z   AWS_REGION: us-east-1
2025-07-17T08:43:51.9947215Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:51.9947818Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:51.9953109Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:51.9953345Z ##[endgroup]
2025-07-17T08:43:52.0000775Z + python3 /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2025-07-17T08:43:52.0186779Z Setting output branch=main
2025-07-17T08:43:52.0248742Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2025-07-17T08:43:52.0249115Z echo "Workflow: ${GITHUB_WORKFLOW}"
2025-07-17T08:43:52.0249427Z echo "Job name: ${JOB_NAME}"
2025-07-17T08:43:52.0249685Z 
2025-07-17T08:43:52.0250023Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-07-17T08:43:52.0250495Z # in runner workspace
2025-07-17T08:43:52.0250886Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2025-07-17T08:43:52.0251340Z   --workflow "${GITHUB_WORKFLOW}" \
2025-07-17T08:43:52.0251669Z   --job-name "${JOB_NAME}" \
2025-07-17T08:43:52.0252347Z   --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}]}" \
2025-07-17T08:43:52.0253041Z   --selected-test-configs "" \
2025-07-17T08:43:52.0253344Z   --pr-number "${PR_NUMBER}" \
2025-07-17T08:43:52.0253619Z   --tag "${TAG}" \
2025-07-17T08:43:52.0253877Z   --event-name "${EVENT_NAME}" \
2025-07-17T08:43:52.0254161Z   --schedule "${SCHEDULE}" \
2025-07-17T08:43:52.0254435Z   --branch "${HEAD_BRANCH}"
2025-07-17T08:43:52.0282021Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:43:52.0282368Z env:
2025-07-17T08:43:52.0282572Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:52.0282945Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:52.0283502Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:52.0284032Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:52.0285129Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:52.0285900Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:52.0286158Z   AWS_REGION: us-east-1
2025-07-17T08:43:52.0286463Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:52.0286793Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:52.0292076Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:52.0292476Z   GITHUB_TOKEN: ***
2025-07-17T08:43:52.0292800Z   JOB_NAME: rocm-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2)
2025-07-17T08:43:52.0293174Z   PR_NUMBER: 
2025-07-17T08:43:52.0293369Z   TAG: 
2025-07-17T08:43:52.0293564Z   EVENT_NAME: push
2025-07-17T08:43:52.0293768Z   SCHEDULE: 
2025-07-17T08:43:52.0293968Z   HEAD_BRANCH: main
2025-07-17T08:43:52.0294181Z ##[endgroup]
2025-07-17T08:43:52.0344400Z Workflow: inductor-rocm
2025-07-17T08:43:52.0344806Z Job name: rocm-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2)
2025-07-17T08:43:52.3742831Z Setting output keep-going=True
2025-07-17T08:43:52.3743359Z Setting output ci-verbose-test-logs=False
2025-07-17T08:43:52.3743910Z Setting output ci-test-showlocals=False
2025-07-17T08:43:52.3744419Z Setting output ci-no-test-timeout=False
2025-07-17T08:43:52.3744904Z Setting output ci-no-td=False
2025-07-17T08:43:52.3745373Z Setting output ci-td-distributed=False
2025-07-17T08:43:52.3745865Z Setting output is-unstable=False
2025-07-17T08:43:52.3746320Z Setting output reenabled-issues=
2025-07-17T08:43:52.3747501Z Setting output test-matrix={"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}]}
2025-07-17T08:43:52.3748551Z Setting output is-test-matrix-empty=False
2025-07-17T08:43:52.3930958Z ##[group]Run echo "Filtered matrix:"
2025-07-17T08:43:52.3931269Z echo "Filtered matrix:"
2025-07-17T08:43:52.3931923Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}]}"
2025-07-17T08:43:52.3932582Z 
2025-07-17T08:43:52.3932757Z echo
2025-07-17T08:43:52.3932992Z echo "Is the current job unstable? False"
2025-07-17T08:43:52.3933287Z 
2025-07-17T08:43:52.3933482Z echo
2025-07-17T08:43:52.3933713Z echo "Is keep-going label set? True"
2025-07-17T08:43:52.3933992Z 
2025-07-17T08:43:52.3934173Z echo
2025-07-17T08:43:52.3934377Z echo "Reenabled issues? "
2025-07-17T08:43:52.3960640Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:43:52.3960977Z env:
2025-07-17T08:43:52.3961193Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:52.3961570Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:52.3962123Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:52.3962631Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:52.3963487Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:52.3964244Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:52.3964487Z   AWS_REGION: us-east-1
2025-07-17T08:43:52.3964778Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:52.3965102Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:52.3970355Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:52.3970582Z ##[endgroup]
2025-07-17T08:43:52.4018900Z Filtered matrix:
2025-07-17T08:43:52.4019480Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.rocm.gpu.2}, {config: inductor, shard: 2, num_shards: 2, runner: linux.rocm.gpu.2}]}
2025-07-17T08:43:52.4019995Z 
2025-07-17T08:43:52.4020099Z Is the current job unstable? False
2025-07-17T08:43:52.4020284Z 
2025-07-17T08:43:52.4020382Z Is keep-going label set? True
2025-07-17T08:43:52.4020552Z 
2025-07-17T08:43:52.4020872Z Reenabled issues? 
2025-07-17T08:43:52.4062770Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-07-17T08:43:52.4063261Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-07-17T08:43:52.4089416Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T08:43:52.4089754Z env:
2025-07-17T08:43:52.4089959Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:52.4090333Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:52.4090870Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:52.4091384Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:52.4092239Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:52.4092992Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:52.4093286Z   AWS_REGION: us-east-1
2025-07-17T08:43:52.4093584Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:52.4093910Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:52.4099253Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:52.4099498Z   JOB_TIMEOUT: 300
2025-07-17T08:43:52.4099705Z ##[endgroup]
2025-07-17T08:43:52.4203978Z ##[group]Run set -x
2025-07-17T08:43:52.4204300Z set -x
2025-07-17T08:43:52.4204509Z 
2025-07-17T08:43:52.4204744Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2025-07-17T08:43:52.4205102Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2025-07-17T08:43:52.4205471Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2025-07-17T08:43:52.4206107Z   TEST_COMMAND=.ci/caffe2/test.sh
2025-07-17T08:43:52.4206411Z else
2025-07-17T08:43:52.4206646Z   TEST_COMMAND=.ci/pytorch/test.sh
2025-07-17T08:43:52.4206929Z fi
2025-07-17T08:43:52.4207120Z 
2025-07-17T08:43:52.4207426Z # detached container should get cleaned up by teardown_ec2_linux
2025-07-17T08:43:52.4207904Z # TODO: Stop building test binaries as part of the build phase
2025-07-17T08:43:52.4208328Z # Used for GPU_FLAG since that doesn't play nice
2025-07-17T08:43:52.4208700Z # shellcheck disable=SC2086,SC2090
2025-07-17T08:43:52.4208994Z container_name=$(docker run \
2025-07-17T08:43:52.4209281Z   ${GPU_FLAG:-} \
2025-07-17T08:43:52.4209544Z   -e BUILD_ENVIRONMENT \
2025-07-17T08:43:52.4209811Z   -e PR_NUMBER \
2025-07-17T08:43:52.4210053Z   -e GITHUB_ACTIONS \
2025-07-17T08:43:52.4210306Z   -e GITHUB_REPOSITORY \
2025-07-17T08:43:52.4210570Z   -e GITHUB_WORKFLOW \
2025-07-17T08:43:52.4210821Z   -e GITHUB_JOB \
2025-07-17T08:43:52.4211053Z   -e GITHUB_RUN_ID \
2025-07-17T08:43:52.4211297Z   -e GITHUB_RUN_NUMBER \
2025-07-17T08:43:52.4211551Z   -e GITHUB_RUN_ATTEMPT \
2025-07-17T08:43:52.4211810Z   -e JOB_ID \
2025-07-17T08:43:52.4212028Z   -e JOB_NAME \
2025-07-17T08:43:52.4212260Z   -e BRANCH \
2025-07-17T08:43:52.4212479Z   -e SHA1 \
2025-07-17T08:43:52.4212705Z   -e AWS_DEFAULT_REGION \
2025-07-17T08:43:52.4212974Z   -e IN_WHEEL_TEST \
2025-07-17T08:43:52.4213218Z   -e SHARD_NUMBER \
2025-07-17T08:43:52.4213468Z   -e TEST_CONFIG \
2025-07-17T08:43:52.4213711Z   -e NUM_TEST_SHARDS \
2025-07-17T08:43:52.4213967Z   -e REENABLED_ISSUES \
2025-07-17T08:43:52.4214233Z   -e CONTINUE_THROUGH_ERROR \
2025-07-17T08:43:52.4214519Z   -e VERBOSE_TEST_LOGS \
2025-07-17T08:43:52.4214766Z   -e TEST_SHOWLOCALS \
2025-07-17T08:43:52.4215015Z   -e NO_TEST_TIMEOUT \
2025-07-17T08:43:52.4215256Z   -e NO_TD \
2025-07-17T08:43:52.4215507Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2025-07-17T08:43:52.4215826Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2025-07-17T08:43:52.4216237Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2025-07-17T08:43:52.4216547Z   -e TESTS_TO_INCLUDE \
2025-07-17T08:43:52.4216814Z   -e DASHBOARD_TAG \
2025-07-17T08:43:52.4217138Z   --env-file="${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}" \
2025-07-17T08:43:52.4217502Z   --ulimit stack=10485760:83886080 \
2025-07-17T08:43:52.4217788Z   --ulimit core=0 \
2025-07-17T08:43:52.4218064Z   --security-opt seccomp=unconfined \
2025-07-17T08:43:52.4218372Z   --cap-add=SYS_PTRACE \
2025-07-17T08:43:52.4218631Z   --shm-size="8g" \
2025-07-17T08:43:52.4218876Z   --tty \
2025-07-17T08:43:52.4219109Z   --detach \
2025-07-17T08:43:52.4219350Z   --name="${container_name}" \
2025-07-17T08:43:52.4219631Z   --user jenkins \
2025-07-17T08:43:52.4219939Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2025-07-17T08:43:52.4220294Z   -w /var/lib/jenkins/workspace \
2025-07-17T08:43:52.4220579Z   "${DOCKER_IMAGE}"
2025-07-17T08:43:52.4220808Z )
2025-07-17T08:43:52.4221027Z # save container name for later step
2025-07-17T08:43:52.4221589Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV"
2025-07-17T08:43:52.4222241Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home
2025-07-17T08:43:52.4223051Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}"
2025-07-17T08:43:52.4250815Z shell: /usr/bin/bash -e {0}
2025-07-17T08:43:52.4251075Z env:
2025-07-17T08:43:52.4251284Z   GIT_DEFAULT_BRANCH: main
2025-07-17T08:43:52.4251664Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T08:43:52.4252240Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T08:43:52.4252759Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T08:43:52.4253610Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T08:43:52.4254371Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T08:43:52.4254627Z   AWS_REGION: us-east-1
2025-07-17T08:43:52.4254932Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T08:43:52.4255268Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T08:43:52.4260636Z   AWS_SESSION_TOKEN: ***
2025-07-17T08:43:52.4260915Z   BUILD_ENVIRONMENT: linux-jammy-rocm-py3.10
2025-07-17T08:43:52.4261208Z   PR_NUMBER: 
2025-07-17T08:43:52.4261439Z   GITHUB_REPOSITORY: pytorch/pytorch
2025-07-17T08:43:52.4261718Z   GITHUB_WORKFLOW: inductor-rocm
2025-07-17T08:43:52.4261962Z   GITHUB_JOB: test
2025-07-17T08:43:52.4262176Z   GITHUB_RUN_ID: 16337959866
2025-07-17T08:43:52.4262416Z   GITHUB_RUN_NUMBER: 38617
2025-07-17T08:43:52.4262650Z   GITHUB_RUN_ATTEMPT: 1
2025-07-17T08:43:52.4262868Z   JOB_ID: 46159470976
2025-07-17T08:43:52.4263184Z   JOB_NAME: rocm-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2)
2025-07-17T08:43:52.4263552Z   BRANCH: main
2025-07-17T08:43:52.4263780Z   SHA1: a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:43:52.4264071Z   CONTINUE_THROUGH_ERROR: True
2025-07-17T08:43:52.4264327Z   VERBOSE_TEST_LOGS: False
2025-07-17T08:43:52.4264565Z   TEST_SHOWLOCALS: False
2025-07-17T08:43:52.4264802Z   NO_TEST_TIMEOUT: False
2025-07-17T08:43:52.4265028Z   NO_TD: False
2025-07-17T08:43:52.4265235Z   TEST_CONFIG: inductor
2025-07-17T08:43:52.4265457Z   SHARD_NUMBER: 1
2025-07-17T08:43:52.4265663Z   NUM_TEST_SHARDS: 2
2025-07-17T08:43:52.4265879Z   REENABLED_ISSUES: 
2025-07-17T08:43:52.4266495Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:43:52.4267163Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2025-07-17T08:43:52.4267443Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2025-07-17T08:43:52.4267695Z   TESTS_TO_INCLUDE: 
2025-07-17T08:43:52.4267904Z   DASHBOARD_TAG: 
2025-07-17T08:43:52.4268115Z ##[endgroup]
2025-07-17T08:43:52.4318011Z + [[ inductor == \m\u\l\t\i\g\p\u ]]
2025-07-17T08:43:52.4318472Z + [[ linux-jammy-rocm-py3.10 == *onnx* ]]
2025-07-17T08:43:52.4319008Z + TEST_COMMAND=.ci/pytorch/test.sh
2025-07-17T08:43:52.4333064Z +++ nproc --ignore=2
2025-07-17T08:43:52.4354264Z ++ docker run --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE -e DASHBOARD_TAG --env-file=/var/home/pytorchci/actions-runner/_work/_temp/github_env_16337959866 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /var/home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-01345e7669bb7198df9fce7a02a4a12ce8c84f2d
2025-07-17T08:43:52.5864160Z + container_name=16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a
2025-07-17T08:43:52.5864931Z + echo CONTAINER_NAME=16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a
2025-07-17T08:43:52.5866113Z + docker exec -t 16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh'
2025-07-17T08:44:13.0267617Z Processing ./dist/torch-2.9.0a0+gita38f433-cp310-cp310-linux_x86_64.whl
2025-07-17T08:44:14.1094743Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+gita38f433) (3.18.0)
2025-07-17T08:44:14.1099061Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+gita38f433) (4.14.1)
2025-07-17T08:44:14.1103378Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+gita38f433) (1.13.3)
2025-07-17T08:44:14.1106488Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+gita38f433) (2.8.8)
2025-07-17T08:44:14.1109670Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+gita38f433) (3.1.6)
2025-07-17T08:44:14.1113090Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+gita38f433) (2025.5.1)
2025-07-17T08:44:14.1127910Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.9.0a0+gita38f433) (1.3.0)
2025-07-17T08:44:14.1493753Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.9.0a0+gita38f433) (3.0.2)
2025-07-17T08:44:14.7863067Z Installing collected packages: torch
2025-07-17T08:44:27.7619096Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2025-07-17T08:44:27.7620716Z helion 0.0.9 requires filecheck, which is not installed.
2025-07-17T08:44:27.7621670Z timm 1.0.14 requires torchvision, which is not installed.
2025-07-17T08:44:27.7622514Z Successfully installed torch-2.9.0a0+gita38f433
2025-07-17T08:44:27.8254472Z + export TERM=vt100
2025-07-17T08:44:27.8254833Z + TERM=vt100
2025-07-17T08:44:27.8261983Z ++ dirname .ci/pytorch/test.sh
2025-07-17T08:44:27.8277368Z + source .ci/pytorch/common.sh
2025-07-17T08:44:27.8285696Z +++ dirname .ci/pytorch/common.sh
2025-07-17T08:44:27.8302827Z ++ source .ci/pytorch/common_utils.sh
2025-07-17T08:44:27.8304312Z +++ declare -f -t trap_add
2025-07-17T08:44:27.8308619Z ++ set -ex -o pipefail
2025-07-17T08:44:27.8308899Z ++ [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-07-17T08:44:27.8309217Z ++ unset HIP_PLATFORM
2025-07-17T08:44:27.8309449Z ++ export PYTORCH_TEST_WITH_ROCM=1
2025-07-17T08:44:27.8309718Z ++ PYTORCH_TEST_WITH_ROCM=1
2025-07-17T08:44:27.8309962Z ++ BUILD_TEST_LIBTORCH=0
2025-07-17T08:44:27.8316073Z ++ dirname .ci/pytorch/test.sh
2025-07-17T08:44:27.8330921Z + source .ci/pytorch/common-build.sh
2025-07-17T08:44:27.8333055Z ++ [[ linux-jammy-rocm-py3.10 != *win-* ]]
2025-07-17T08:44:27.8346775Z ++++ dirname .ci/pytorch/common-build.sh
2025-07-17T08:44:27.8362663Z +++ cd .ci/pytorch
2025-07-17T08:44:27.8363299Z +++ pwd -P
2025-07-17T08:44:27.8367471Z ++ script_dir=/var/lib/jenkins/pytorch/.ci/pytorch
2025-07-17T08:44:27.8368502Z ++ [[ linux-jammy-rocm-py3.10 == *-pch* ]]
2025-07-17T08:44:27.8368804Z ++ which sccache
2025-07-17T08:44:27.8387632Z ++ [[ -z '' ]]
2025-07-17T08:44:27.8387840Z ++ unset SCCACHE_BUCKET
2025-07-17T08:44:27.8388194Z ++ unset SCCACHE_REGION
2025-07-17T08:44:27.8388529Z ++ sccache --stop-server
2025-07-17T08:44:27.8426663Z ++ true
2025-07-17T08:44:27.8427386Z ++ rm -f /var/lib/jenkins/sccache_error.log
2025-07-17T08:44:27.8443238Z ++ trap_add sccache_epilogue EXIT
2025-07-17T08:44:27.8443653Z ++ trap_add_cmd=sccache_epilogue
2025-07-17T08:44:27.8444022Z ++ shift
2025-07-17T08:44:27.8444896Z ++ for trap_add_name in "$@"
2025-07-17T08:44:27.8460849Z ++++ trap -p EXIT
2025-07-17T08:44:27.8461177Z +++ eval 'extract_trap_cmd '
2025-07-17T08:44:27.8461437Z ++++ extract_trap_cmd
2025-07-17T08:44:27.8461656Z ++++ printf '%s\n' ''
2025-07-17T08:44:27.8461878Z +++ printf '%s\n' sccache_epilogue
2025-07-17T08:44:27.8463571Z ++ trap -- '
2025-07-17T08:44:27.8463780Z sccache_epilogue' EXIT
2025-07-17T08:44:27.8464081Z ++ [[ -n '' ]]
2025-07-17T08:44:27.8464301Z ++ [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-07-17T08:44:27.8464630Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2025-07-17T08:44:27.8464930Z ++ SCCACHE_IDLE_TIMEOUT=0
2025-07-17T08:44:27.8465159Z ++ sccache --start-server
2025-07-17T08:44:27.8495166Z sccache: Starting the server...
2025-07-17T08:44:27.8861519Z sccache: Listening on address 127.0.0.1:4226
2025-07-17T08:44:27.8880838Z ++ sccache --zero-stats
2025-07-17T08:44:27.8910593Z Statistics zeroed.
2025-07-17T08:44:27.8916114Z ++ which ccache
2025-07-17T08:44:27.8931094Z + [[ linux-jammy-rocm-py3.10 != *rocm* ]]
2025-07-17T08:44:27.8931453Z + echo 'Environment variables:'
2025-07-17T08:44:27.8931729Z Environment variables:
2025-07-17T08:44:27.8931948Z + env
2025-07-17T08:44:27.8944362Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-07-17T08:44:27.8944792Z CONTINUE_THROUGH_ERROR=True
2025-07-17T08:44:27.8945091Z BUILD_ENVIRONMENT=linux-jammy-rocm-py3.10
2025-07-17T08:44:27.8945432Z HOSTNAME=gpu4f67.jax.cs.cpe.ice.amd.com
2025-07-17T08:44:27.8946048Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_c87b372e-99dd-4da0-8cc0-d8099715a65d
2025-07-17T08:44:27.8946620Z GITHUB_ACTION=__self
2025-07-17T08:44:27.8946856Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-07-17T08:44:27.8947119Z GITHUB_RUN_NUMBER=38617
2025-07-17T08:44:27.8947353Z TEST_CONFIG=inductor
2025-07-17T08:44:27.8947579Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-07-17T08:44:27.8947860Z AWS_DEFAULT_REGION=us-east-1
2025-07-17T08:44:27.8948124Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2025-07-17T08:44:27.8948398Z GITHUB_REF_TYPE=branch
2025-07-17T08:44:27.8948840Z ***
2025-07-17T08:44:27.8949047Z GITHUB_REPOSITORY_ID=65600975
2025-07-17T08:44:27.8949304Z GITHUB_ACTIONS=true
2025-07-17T08:44:27.8949539Z SHA1=a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:44:27.8949872Z GITHUB_SHA=a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:44:27.8950365Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-rocm.yml@refs/heads/main
2025-07-17T08:44:27.8950803Z UCC_HOME=/usr
2025-07-17T08:44:27.8951003Z VERBOSE_TEST_LOGS=False
2025-07-17T08:44:27.8951232Z GITHUB_REF=refs/heads/main
2025-07-17T08:44:27.8951463Z SHARD_NUMBER=1
2025-07-17T08:44:27.8951674Z GITHUB_REF_PROTECTED=true
2025-07-17T08:44:27.8951904Z HOME=/var/lib/jenkins
2025-07-17T08:44:27.8952165Z GITHUB_API_URL=https://api.github.com
2025-07-17T08:44:27.8952467Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-07-17T08:44:27.8952722Z LANG=C.UTF-8
2025-07-17T08:44:27.8952967Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6
2025-07-17T08:44:27.8953283Z PYTORCH_TEST_WITH_ROCM=1
2025-07-17T08:44:27.8953524Z NUM_TEST_SHARDS=2
2025-07-17T08:44:27.8953727Z UCX_HOME=/usr
2025-07-17T08:44:27.8954296Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_c87b372e-99dd-4da0-8cc0-d8099715a65d
2025-07-17T08:44:27.8964076Z JOB_NAME=rocm-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2)
2025-07-17T08:44:27.8964483Z MAGMA_HOME=/opt/rocm/magma
2025-07-17T08:44:27.8965013Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_c87b372e-99dd-4da0-8cc0-d8099715a65d
2025-07-17T08:44:27.8965746Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2025-07-17T08:44:27.8966187Z GITHUB_EVENT_NAME=push
2025-07-17T08:44:27.8966607Z DASHBOARD_TAG=
2025-07-17T08:44:27.8966805Z GITHUB_RUN_ID=16337959866
2025-07-17T08:44:27.8967371Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_c87b372e-99dd-4da0-8cc0-d8099715a65d
2025-07-17T08:44:27.8967964Z GITHUB_ACTOR=pytorchmergebot
2025-07-17T08:44:27.8968186Z PR_NUMBER=
2025-07-17T08:44:27.8968371Z GITHUB_RUN_ATTEMPT=1
2025-07-17T08:44:27.8968588Z ANACONDA_PYTHON_VERSION=3.10
2025-07-17T08:44:27.8968869Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-07-17T08:44:27.8969163Z TERM=vt100
2025-07-17T08:44:27.8969350Z INSTALLED_VISION=yes
2025-07-17T08:44:27.8969551Z BRANCH=main
2025-07-17T08:44:27.8969747Z OPENSSL_ROOT_DIR=/opt/openssl
2025-07-17T08:44:27.8969980Z TESTS_TO_INCLUDE=
2025-07-17T08:44:27.8970416Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-07-17T08:44:27.8970924Z GITHUB_SERVER_URL=https://github.com
2025-07-17T08:44:27.8971208Z PYTORCH_ROCM_ARCH=gfx90a;gfx942
2025-07-17T08:44:27.8971491Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d
2025-07-17T08:44:27.8971777Z REENABLED_ISSUES=
2025-07-17T08:44:27.8971969Z SHLVL=1
2025-07-17T08:44:27.8972141Z MAX_JOBS=126
2025-07-17T08:44:27.8972340Z GITHUB_ACTOR_ID=97764156
2025-07-17T08:44:27.8972632Z GITHUB_WORKFLOW_SHA=a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:44:27.8972956Z GITHUB_REF_NAME=main
2025-07-17T08:44:27.8973161Z ROCM_PATH=/opt/rocm
2025-07-17T08:44:27.8973352Z GITHUB_JOB=test
2025-07-17T08:44:27.8973549Z NO_TEST_TIMEOUT=False
2025-07-17T08:44:27.8973777Z GITHUB_REPOSITORY=pytorch/pytorch
2025-07-17T08:44:27.8974023Z LC_ALL=C.UTF-8
2025-07-17T08:44:27.8974224Z GITHUB_RETENTION_DAYS=90
2025-07-17T08:44:27.8974448Z OPENSSL_DIR=/opt/openssl
2025-07-17T08:44:27.8974674Z GITHUB_ACTION_REPOSITORY=
2025-07-17T08:44:27.8975484Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-07-17T08:44:27.8976404Z GITHUB_BASE_REF=
2025-07-17T08:44:27.8976593Z CI=true
2025-07-17T08:44:27.8976783Z GITHUB_REPOSITORY_OWNER=pytorch
2025-07-17T08:44:27.8977019Z JOB_ID=46159470976
2025-07-17T08:44:27.8977214Z GITHUB_HEAD_REF=
2025-07-17T08:44:27.8977405Z GITHUB_ACTION_REF=
2025-07-17T08:44:27.8977607Z TEST_SHOWLOCALS=False
2025-07-17T08:44:27.8977831Z GITHUB_WORKFLOW=inductor-rocm
2025-07-17T08:44:27.8978084Z DEBIAN_FRONTEND=noninteractive
2025-07-17T08:44:27.8978651Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_c87b372e-99dd-4da0-8cc0-d8099715a65d
2025-07-17T08:44:27.8979209Z NO_TD=False
2025-07-17T08:44:27.8979398Z OLDPWD=/var/lib/jenkins
2025-07-17T08:44:27.8979612Z _=/usr/bin/env
2025-07-17T08:44:27.8979879Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2025-07-17T08:44:27.9093566Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2025-07-17T08:44:27.9094492Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-07-17T08:44:27.9095313Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2025-07-17T08:44:27.9096299Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2025-07-17T08:44:27.9096948Z + BUILD_DIR=build
2025-07-17T08:44:27.9097337Z + BUILD_RENAMED_DIR=build_renamed
2025-07-17T08:44:27.9097762Z + BUILD_BIN_DIR=build/bin
2025-07-17T08:44:27.9098132Z + SHARD_NUMBER=1
2025-07-17T08:44:27.9098457Z + NUM_TEST_SHARDS=2
2025-07-17T08:44:27.9099132Z + export TORCH_SERIALIZATION_DEBUG=1
2025-07-17T08:44:27.9099602Z + TORCH_SERIALIZATION_DEBUG=1
2025-07-17T08:44:27.9100016Z + export VALGRIND=ON
2025-07-17T08:44:27.9100358Z + VALGRIND=ON
2025-07-17T08:44:27.9100782Z + [[ linux-jammy-rocm-py3.10 == *clang9* ]]
2025-07-17T08:44:27.9101367Z + [[ linux-jammy-rocm-py3.10 == *xpu* ]]
2025-07-17T08:44:27.9101933Z + [[ linux-jammy-rocm-py3.10 == *s390x* ]]
2025-07-17T08:44:27.9102841Z + [[ 0 == \1 ]]
2025-07-17T08:44:27.9103208Z + [[ True == \1 ]]
2025-07-17T08:44:27.9103638Z + [[ linux-jammy-rocm-py3.10 != *bazel* ]]
2025-07-17T08:44:27.9104218Z ++ realpath build/custom_test_artifacts
2025-07-17T08:44:27.9116930Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts
2025-07-17T08:44:27.9117342Z + [[ -n '' ]]
2025-07-17T08:44:27.9117557Z + echo 'Environment variables'
2025-07-17T08:44:27.9117799Z Environment variables
2025-07-17T08:44:27.9118001Z + env
2025-07-17T08:44:27.9134495Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-07-17T08:44:27.9134886Z CONTINUE_THROUGH_ERROR=True
2025-07-17T08:44:27.9135171Z BUILD_ENVIRONMENT=linux-jammy-rocm-py3.10
2025-07-17T08:44:27.9135470Z HOSTNAME=gpu4f67.jax.cs.cpe.ice.amd.com
2025-07-17T08:44:27.9136109Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_c87b372e-99dd-4da0-8cc0-d8099715a65d
2025-07-17T08:44:27.9136670Z GITHUB_ACTION=__self
2025-07-17T08:44:27.9136901Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-07-17T08:44:27.9137157Z GITHUB_RUN_NUMBER=38617
2025-07-17T08:44:27.9137370Z TEST_CONFIG=inductor
2025-07-17T08:44:27.9137592Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-07-17T08:44:27.9137855Z AWS_DEFAULT_REGION=us-east-1
2025-07-17T08:44:27.9138110Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2025-07-17T08:44:27.9138401Z GITHUB_REF_TYPE=branch
2025-07-17T08:44:27.9138660Z ***
2025-07-17T08:44:27.9138863Z GITHUB_REPOSITORY_ID=65600975
2025-07-17T08:44:27.9139106Z GITHUB_ACTIONS=true
2025-07-17T08:44:27.9139338Z SHA1=a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:44:27.9139652Z GITHUB_SHA=a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:44:27.9140106Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-rocm.yml@refs/heads/main
2025-07-17T08:44:27.9140515Z UCC_HOME=/usr
2025-07-17T08:44:27.9140718Z TORCH_SERIALIZATION_DEBUG=1
2025-07-17T08:44:27.9140955Z VERBOSE_TEST_LOGS=False
2025-07-17T08:44:27.9141190Z GITHUB_REF=refs/heads/main
2025-07-17T08:44:27.9141415Z SHARD_NUMBER=1
2025-07-17T08:44:27.9141616Z GITHUB_REF_PROTECTED=true
2025-07-17T08:44:27.9141836Z HOME=/var/lib/jenkins
2025-07-17T08:44:27.9142192Z GITHUB_API_URL=https://api.github.com
2025-07-17T08:44:27.9142482Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-07-17T08:44:27.9142727Z LANG=C.UTF-8
2025-07-17T08:44:27.9142966Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6
2025-07-17T08:44:27.9143277Z PYTORCH_TEST_WITH_ROCM=1
2025-07-17T08:44:27.9143502Z NUM_TEST_SHARDS=2
2025-07-17T08:44:27.9143697Z UCX_HOME=/usr
2025-07-17T08:44:27.9144223Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_c87b372e-99dd-4da0-8cc0-d8099715a65d
2025-07-17T08:44:27.9144918Z JOB_NAME=rocm-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2)
2025-07-17T08:44:27.9145294Z MAGMA_HOME=/opt/rocm/magma
2025-07-17T08:44:27.9145831Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_c87b372e-99dd-4da0-8cc0-d8099715a65d
2025-07-17T08:44:27.9146600Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2025-07-17T08:44:27.9147060Z GITHUB_EVENT_NAME=push
2025-07-17T08:44:27.9147275Z DASHBOARD_TAG=
2025-07-17T08:44:27.9147473Z GITHUB_RUN_ID=16337959866
2025-07-17T08:44:27.9148056Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_c87b372e-99dd-4da0-8cc0-d8099715a65d
2025-07-17T08:44:27.9148687Z GITHUB_ACTOR=pytorchmergebot
2025-07-17T08:44:27.9148933Z PR_NUMBER=
2025-07-17T08:44:27.9149121Z GITHUB_RUN_ATTEMPT=1
2025-07-17T08:44:27.9149328Z VALGRIND=ON
2025-07-17T08:44:27.9149713Z ANACONDA_PYTHON_VERSION=3.10
2025-07-17T08:44:27.9150015Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-07-17T08:44:27.9150321Z TERM=vt100
2025-07-17T08:44:27.9150514Z INSTALLED_VISION=yes
2025-07-17T08:44:27.9150717Z BRANCH=main
2025-07-17T08:44:27.9150913Z OPENSSL_ROOT_DIR=/opt/openssl
2025-07-17T08:44:27.9151156Z TESTS_TO_INCLUDE=
2025-07-17T08:44:27.9151767Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-07-17T08:44:27.9152267Z GITHUB_SERVER_URL=https://github.com
2025-07-17T08:44:27.9152539Z PYTORCH_ROCM_ARCH=gfx90a;gfx942
2025-07-17T08:44:27.9152817Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d
2025-07-17T08:44:27.9153104Z REENABLED_ISSUES=
2025-07-17T08:44:27.9153289Z SHLVL=1
2025-07-17T08:44:27.9153460Z MAX_JOBS=126
2025-07-17T08:44:27.9153650Z GITHUB_ACTOR_ID=97764156
2025-07-17T08:44:27.9153934Z GITHUB_WORKFLOW_SHA=a38f433be2e94a64b095a44ba39879d02d0c2316
2025-07-17T08:44:27.9154261Z GITHUB_REF_NAME=main
2025-07-17T08:44:27.9154468Z ROCM_PATH=/opt/rocm
2025-07-17T08:44:27.9154666Z GITHUB_JOB=test
2025-07-17T08:44:27.9154860Z NO_TEST_TIMEOUT=False
2025-07-17T08:44:27.9155085Z GITHUB_REPOSITORY=pytorch/pytorch
2025-07-17T08:44:27.9155325Z LC_ALL=C.UTF-8
2025-07-17T08:44:27.9155517Z GITHUB_RETENTION_DAYS=90
2025-07-17T08:44:27.9155744Z OPENSSL_DIR=/opt/openssl
2025-07-17T08:44:27.9155969Z GITHUB_ACTION_REPOSITORY=
2025-07-17T08:44:27.9156773Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-07-17T08:44:27.9157583Z GITHUB_BASE_REF=
2025-07-17T08:44:27.9157765Z CI=true
2025-07-17T08:44:27.9157953Z GITHUB_REPOSITORY_OWNER=pytorch
2025-07-17T08:44:27.9158188Z JOB_ID=46159470976
2025-07-17T08:44:27.9158378Z GITHUB_HEAD_REF=
2025-07-17T08:44:27.9158569Z GITHUB_ACTION_REF=
2025-07-17T08:44:27.9158766Z TEST_SHOWLOCALS=False
2025-07-17T08:44:27.9158990Z GITHUB_WORKFLOW=inductor-rocm
2025-07-17T08:44:27.9159237Z DEBIAN_FRONTEND=noninteractive
2025-07-17T08:44:27.9159789Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_c87b372e-99dd-4da0-8cc0-d8099715a65d
2025-07-17T08:44:27.9160339Z NO_TD=False
2025-07-17T08:44:27.9160520Z OLDPWD=/var/lib/jenkins
2025-07-17T08:44:27.9160730Z _=/usr/bin/env
2025-07-17T08:44:27.9160936Z + echo 'Testing pytorch'
2025-07-17T08:44:27.9161151Z Testing pytorch
2025-07-17T08:44:27.9161350Z + export LANG=C.UTF-8
2025-07-17T08:44:27.9161550Z + LANG=C.UTF-8
2025-07-17T08:44:27.9161732Z + PR_NUMBER=
2025-07-17T08:44:27.9161928Z + [[ inductor == \d\e\f\a\u\l\t ]]
2025-07-17T08:44:27.9162187Z + [[ inductor == \d\i\s\t\r\i\b\u\t\e\d ]]
2025-07-17T08:44:27.9162451Z + [[ inductor == \s\l\o\w ]]
2025-07-17T08:44:27.9162713Z + [[ linux-jammy-rocm-py3.10 == *slow-gradcheck* ]]
2025-07-17T08:44:27.9163028Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]]
2025-07-17T08:44:27.9163313Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-07-17T08:44:27.9163601Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-07-17T08:44:27.9163896Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-07-17T08:44:27.9164165Z + [[ inductor == *crossref* ]]
2025-07-17T08:44:27.9164419Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-07-17T08:44:27.9164676Z + export VALGRIND=OFF
2025-07-17T08:44:27.9164879Z + VALGRIND=OFF
2025-07-17T08:44:27.9165064Z + rocminfo
2025-07-17T08:44:27.9276105Z ROCk module version 6.10.5 is loaded
2025-07-17T08:44:28.1434645Z =====================    
2025-07-17T08:44:28.1434968Z HSA System Attributes    
2025-07-17T08:44:28.1435298Z =====================    
2025-07-17T08:44:28.1435760Z Runtime Version:         1.15
2025-07-17T08:44:28.1436381Z Runtime Ext Version:     1.7
2025-07-17T08:44:28.1437023Z System Timestamp Freq.:  1000.000000MHz
2025-07-17T08:44:28.1438037Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-07-17T08:44:28.1439190Z Machine Model:           LARGE                              
2025-07-17T08:44:28.1439840Z System Endianness:       LITTLE                             
2025-07-17T08:44:28.1440383Z Mwaitx:                  DISABLED
2025-07-17T08:44:28.1440790Z XNACK enabled:           NO
2025-07-17T08:44:28.1441187Z DMAbuf Support:          YES
2025-07-17T08:44:28.1441558Z VMM Support:             YES
2025-07-17T08:44:28.1442089Z 
2025-07-17T08:44:28.1442228Z ==========               
2025-07-17T08:44:28.1442595Z HSA Agents               
2025-07-17T08:44:28.1442945Z ==========               
2025-07-17T08:44:28.1443277Z *******                  
2025-07-17T08:44:28.1443617Z Agent 1                  
2025-07-17T08:44:28.1443965Z *******                  
2025-07-17T08:44:28.1444405Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-07-17T08:44:28.1444981Z   Uuid:                    CPU-XX                             
2025-07-17T08:44:28.1445759Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-17T08:44:28.1446610Z   Vendor Name:             CPU                                
2025-07-17T08:44:28.1447414Z   Feature:                 None specified                     
2025-07-17T08:44:28.1448213Z   Profile:                 FULL_PROFILE                       
2025-07-17T08:44:28.1449031Z   Float Round Mode:        NEAR                               
2025-07-17T08:44:28.1449533Z   Max Queue Number:        0(0x0)                             
2025-07-17T08:44:28.1449977Z   Queue Min Size:          0(0x0)                             
2025-07-17T08:44:28.1450385Z   Queue Max Size:          0(0x0)                             
2025-07-17T08:44:28.1450790Z   Queue Type:              MULTI                              
2025-07-17T08:44:28.1451156Z   Node:                    0                                  
2025-07-17T08:44:28.1451528Z   Device Type:             CPU                                
2025-07-17T08:44:28.1451891Z   Cache Info:              
2025-07-17T08:44:28.1452215Z     L1:                      32768(0x8000) KB                   
2025-07-17T08:44:28.1452591Z   Chip ID:                 0(0x0)                             
2025-07-17T08:44:28.1452996Z   ASIC Revision:           0(0x0)                             
2025-07-17T08:44:28.1453411Z   Cacheline Size:          64(0x40)                           
2025-07-17T08:44:28.1453946Z   Max Clock Freq. (MHz):   2000                               
2025-07-17T08:44:28.1454346Z   BDFID:                   0                                  
2025-07-17T08:44:28.1454731Z   Internal Node ID:        0                                  
2025-07-17T08:44:28.1455132Z   Compute Unit:            64                                 
2025-07-17T08:44:28.1455531Z   SIMDs per CU:            0                                  
2025-07-17T08:44:28.1456010Z   Shader Engines:          0                                  
2025-07-17T08:44:28.1456452Z   Shader Arrs. per Eng.:   0                                  
2025-07-17T08:44:28.1456807Z   WatchPts on Addr. Ranges:1                                  
2025-07-17T08:44:28.1457120Z   Memory Properties:       
2025-07-17T08:44:28.1457362Z   Features:                None
2025-07-17T08:44:28.1457611Z   Pool Info:               
2025-07-17T08:44:28.1457835Z     Pool 1                   
2025-07-17T08:44:28.1458122Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:44:28.1458468Z       Size:                    528249800(0x1f7c73c8) KB           
2025-07-17T08:44:28.1458790Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1459145Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1459525Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:44:28.1459901Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1460258Z       Accessible by all:       TRUE                               
2025-07-17T08:44:28.1460596Z     Pool 2                   
2025-07-17T08:44:28.1461032Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:44:28.1461375Z       Size:                    528249800(0x1f7c73c8) KB           
2025-07-17T08:44:28.1461709Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1462060Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1462560Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:44:28.1462928Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1463296Z       Accessible by all:       TRUE                               
2025-07-17T08:44:28.1463590Z     Pool 3                   
2025-07-17T08:44:28.1463875Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-07-17T08:44:28.1464209Z       Size:                    528249800(0x1f7c73c8) KB           
2025-07-17T08:44:28.1464534Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1464883Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1465249Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:44:28.1465605Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1465964Z       Accessible by all:       TRUE                               
2025-07-17T08:44:28.1466275Z     Pool 4                   
2025-07-17T08:44:28.1466552Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:44:28.1466872Z       Size:                    528249800(0x1f7c73c8) KB           
2025-07-17T08:44:28.1467188Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1467524Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1467873Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:44:28.1468229Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1468574Z       Accessible by all:       TRUE                               
2025-07-17T08:44:28.1468868Z   ISA Info:                
2025-07-17T08:44:28.1469086Z *******                  
2025-07-17T08:44:28.1469294Z Agent 2                  
2025-07-17T08:44:28.1469509Z *******                  
2025-07-17T08:44:28.1469763Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-07-17T08:44:28.1470087Z   Uuid:                    CPU-XX                             
2025-07-17T08:44:28.1470421Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-17T08:44:28.1470775Z   Vendor Name:             CPU                                
2025-07-17T08:44:28.1471104Z   Feature:                 None specified                     
2025-07-17T08:44:28.1471434Z   Profile:                 FULL_PROFILE                       
2025-07-17T08:44:28.1471768Z   Float Round Mode:        NEAR                               
2025-07-17T08:44:28.1472108Z   Max Queue Number:        0(0x0)                             
2025-07-17T08:44:28.1472448Z   Queue Min Size:          0(0x0)                             
2025-07-17T08:44:28.1472776Z   Queue Max Size:          0(0x0)                             
2025-07-17T08:44:28.1473113Z   Queue Type:              MULTI                              
2025-07-17T08:44:28.1473422Z   Node:                    1                                  
2025-07-17T08:44:28.1473738Z   Device Type:             CPU                                
2025-07-17T08:44:28.1474030Z   Cache Info:              
2025-07-17T08:44:28.1474273Z     L1:                      32768(0x8000) KB                   
2025-07-17T08:44:28.1474577Z   Chip ID:                 0(0x0)                             
2025-07-17T08:44:28.1474902Z   ASIC Revision:           0(0x0)                             
2025-07-17T08:44:28.1475254Z   Cacheline Size:          64(0x40)                           
2025-07-17T08:44:28.1475606Z   Max Clock Freq. (MHz):   2000                               
2025-07-17T08:44:28.1476061Z   BDFID:                   0                                  
2025-07-17T08:44:28.1476413Z   Internal Node ID:        1                                  
2025-07-17T08:44:28.1476767Z   Compute Unit:            64                                 
2025-07-17T08:44:28.1477104Z   SIMDs per CU:            0                                  
2025-07-17T08:44:28.1477440Z   Shader Engines:          0                                  
2025-07-17T08:44:28.1477929Z   Shader Arrs. per Eng.:   0                                  
2025-07-17T08:44:28.1478291Z   WatchPts on Addr. Ranges:1                                  
2025-07-17T08:44:28.1478611Z   Memory Properties:       
2025-07-17T08:44:28.1478856Z   Features:                None
2025-07-17T08:44:28.1479105Z   Pool Info:               
2025-07-17T08:44:28.1479330Z     Pool 1                   
2025-07-17T08:44:28.1479619Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:44:28.1479971Z       Size:                    528402424(0x1f7ec7f8) KB           
2025-07-17T08:44:28.1480294Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1480644Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1481009Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:44:28.1481379Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1481738Z       Accessible by all:       TRUE                               
2025-07-17T08:44:28.1482048Z     Pool 2                   
2025-07-17T08:44:28.1482333Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:44:28.1482663Z       Size:                    528402424(0x1f7ec7f8) KB           
2025-07-17T08:44:28.1482991Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1483337Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1483691Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:44:28.1484047Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1484393Z       Accessible by all:       TRUE                               
2025-07-17T08:44:28.1484696Z     Pool 3                   
2025-07-17T08:44:28.1484960Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-07-17T08:44:28.1485298Z       Size:                    528402424(0x1f7ec7f8) KB           
2025-07-17T08:44:28.1485639Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1485983Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1486340Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:44:28.1486698Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1487050Z       Accessible by all:       TRUE                               
2025-07-17T08:44:28.1487352Z     Pool 4                   
2025-07-17T08:44:28.1487625Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:44:28.1487948Z       Size:                    528402424(0x1f7ec7f8) KB           
2025-07-17T08:44:28.1488275Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1488617Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1488976Z       Alloc Recommended Granule:4KB                                
2025-07-17T08:44:28.1489342Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1489687Z       Accessible by all:       TRUE                               
2025-07-17T08:44:28.1489995Z   ISA Info:                
2025-07-17T08:44:28.1490216Z *******                  
2025-07-17T08:44:28.1490437Z Agent 3                  
2025-07-17T08:44:28.1490646Z *******                  
2025-07-17T08:44:28.1490910Z   Name:                    gfx90a                             
2025-07-17T08:44:28.1491231Z   Uuid:                    GPU-1635b42468671753               
2025-07-17T08:44:28.1491712Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.1492069Z   Vendor Name:             AMD                                
2025-07-17T08:44:28.1492412Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:44:28.1492752Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:44:28.1493250Z   Float Round Mode:        NEAR                               
2025-07-17T08:44:28.1493598Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:44:28.1493931Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:44:28.1494271Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:44:28.1494602Z   Queue Type:              MULTI                              
2025-07-17T08:44:28.1494915Z   Node:                    2                                  
2025-07-17T08:44:28.1495226Z   Device Type:             GPU                                
2025-07-17T08:44:28.1495534Z   Cache Info:              
2025-07-17T08:44:28.1495783Z     L1:                      16(0x10) KB                        
2025-07-17T08:44:28.1496157Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:44:28.1496458Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:44:28.1496783Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:44:28.1497133Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:44:28.1497470Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:44:28.1497790Z   BDFID:                   12800                              
2025-07-17T08:44:28.1498106Z   Internal Node ID:        2                                  
2025-07-17T08:44:28.1498439Z   Compute Unit:            104                                
2025-07-17T08:44:28.1498767Z   SIMDs per CU:            4                                  
2025-07-17T08:44:28.1499105Z   Shader Engines:          8                                  
2025-07-17T08:44:28.1499450Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:44:28.1499806Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:44:28.1500165Z   Coherent Host Access:    FALSE                              
2025-07-17T08:44:28.1500481Z   Memory Properties:       
2025-07-17T08:44:28.1500742Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:44:28.1501059Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:44:28.1501407Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:44:28.1501756Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1502078Z   Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1502340Z     x                        1024(0x400)                        
2025-07-17T08:44:28.1502622Z     y                        1024(0x400)                        
2025-07-17T08:44:28.1502903Z     z                        1024(0x400)                        
2025-07-17T08:44:28.1503207Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:44:28.1503545Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:44:28.1503886Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1504187Z   Grid Max Size per Dimension:
2025-07-17T08:44:28.1504433Z     x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1504718Z     y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1505000Z     z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1505319Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:44:28.1505725Z   Packet Processor uCode:: 92                                 
2025-07-17T08:44:28.1506095Z   SDMA engine uCode::      9                                  
2025-07-17T08:44:28.1506601Z   IOMMU Support::          None                               
2025-07-17T08:44:28.1506899Z   Pool Info:               
2025-07-17T08:44:28.1507133Z     Pool 1                   
2025-07-17T08:44:28.1507423Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:44:28.1507760Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1508093Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1508594Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1508952Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1509308Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1509660Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1509965Z     Pool 2                   
2025-07-17T08:44:28.1510239Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:44:28.1510574Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1510897Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1511237Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1511586Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1511940Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1512294Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1512596Z     Pool 3                   
2025-07-17T08:44:28.1512863Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:44:28.1513188Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1513516Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1513867Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1514225Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1514587Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1514940Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1515241Z     Pool 4                   
2025-07-17T08:44:28.1515502Z       Segment:                 GROUP                              
2025-07-17T08:44:28.1515812Z       Size:                    64(0x40) KB                        
2025-07-17T08:44:28.1516141Z       Allocatable:             FALSE                              
2025-07-17T08:44:28.1516477Z       Alloc Granule:           0KB                                
2025-07-17T08:44:28.1516838Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:44:28.1517204Z       Alloc Alignment:         0KB                                
2025-07-17T08:44:28.1517574Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1517875Z   ISA Info:                
2025-07-17T08:44:28.1518090Z     ISA 1                    
2025-07-17T08:44:28.1518371Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:44:28.1518746Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:44:28.1519102Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:44:28.1519452Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1519834Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1520181Z       Fast f16:                TRUE                               
2025-07-17T08:44:28.1520516Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1520852Z       Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1521145Z         x                        1024(0x400)                        
2025-07-17T08:44:28.1521443Z         y                        1024(0x400)                        
2025-07-17T08:44:28.1521870Z         z                        1024(0x400)                        
2025-07-17T08:44:28.1522201Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1522518Z       Grid Max Size per Dimension:
2025-07-17T08:44:28.1522783Z         x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1523077Z         y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1523497Z         z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1523822Z       FBarrier Max Size:       32                                 
2025-07-17T08:44:28.1524126Z *******                  
2025-07-17T08:44:28.1524344Z Agent 4                  
2025-07-17T08:44:28.1524557Z *******                  
2025-07-17T08:44:28.1524800Z   Name:                    gfx90a                             
2025-07-17T08:44:28.1525119Z   Uuid:                    GPU-8cfe31f8885bf623               
2025-07-17T08:44:28.1525472Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.1525817Z   Vendor Name:             AMD                                
2025-07-17T08:44:28.1526153Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:44:28.1526489Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:44:28.1526830Z   Float Round Mode:        NEAR                               
2025-07-17T08:44:28.1527179Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:44:28.1527517Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:44:28.1527850Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:44:28.1528180Z   Queue Type:              MULTI                              
2025-07-17T08:44:28.1528494Z   Node:                    3                                  
2025-07-17T08:44:28.1528805Z   Device Type:             GPU                                
2025-07-17T08:44:28.1529100Z   Cache Info:              
2025-07-17T08:44:28.1529355Z     L1:                      16(0x10) KB                        
2025-07-17T08:44:28.1529653Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:44:28.1529956Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:44:28.1530279Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:44:28.1530619Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:44:28.1530952Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:44:28.1531262Z   BDFID:                   13568                              
2025-07-17T08:44:28.1531575Z   Internal Node ID:        3                                  
2025-07-17T08:44:28.1531900Z   Compute Unit:            104                                
2025-07-17T08:44:28.1532222Z   SIMDs per CU:            4                                  
2025-07-17T08:44:28.1532545Z   Shader Engines:          8                                  
2025-07-17T08:44:28.1532886Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:44:28.1533227Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:44:28.1533577Z   Coherent Host Access:    FALSE                              
2025-07-17T08:44:28.1533885Z   Memory Properties:       
2025-07-17T08:44:28.1534132Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:44:28.1534444Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:44:28.1534781Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:44:28.1535123Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1535438Z   Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1535707Z     x                        1024(0x400)                        
2025-07-17T08:44:28.1536058Z     y                        1024(0x400)                        
2025-07-17T08:44:28.1536326Z     z                        1024(0x400)                        
2025-07-17T08:44:28.1536790Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:44:28.1537135Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:44:28.1537470Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1537765Z   Grid Max Size per Dimension:
2025-07-17T08:44:28.1538009Z     x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1538440Z     y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1538717Z     z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1539028Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:44:28.1539390Z   Packet Processor uCode:: 92                                 
2025-07-17T08:44:28.1539737Z   SDMA engine uCode::      9                                  
2025-07-17T08:44:28.1540073Z   IOMMU Support::          None                               
2025-07-17T08:44:28.1540362Z   Pool Info:               
2025-07-17T08:44:28.1540584Z     Pool 1                   
2025-07-17T08:44:28.1540855Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:44:28.1541185Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1541503Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1541839Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1542192Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1542549Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1542893Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1543192Z     Pool 2                   
2025-07-17T08:44:28.1543462Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:44:28.1543789Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1544113Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1544453Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1544802Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1545163Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1545506Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1545811Z     Pool 3                   
2025-07-17T08:44:28.1546082Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:44:28.1546401Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1546723Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1547060Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1547402Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1547761Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1548106Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1548412Z     Pool 4                   
2025-07-17T08:44:28.1548670Z       Segment:                 GROUP                              
2025-07-17T08:44:28.1548977Z       Size:                    64(0x40) KB                        
2025-07-17T08:44:28.1549298Z       Allocatable:             FALSE                              
2025-07-17T08:44:28.1549641Z       Alloc Granule:           0KB                                
2025-07-17T08:44:28.1549989Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:44:28.1550339Z       Alloc Alignment:         0KB                                
2025-07-17T08:44:28.1550677Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1550975Z   ISA Info:                
2025-07-17T08:44:28.1551186Z     ISA 1                    
2025-07-17T08:44:28.1551618Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:44:28.1551992Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:44:28.1552343Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:44:28.1552692Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1553173Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1553503Z       Fast f16:                TRUE                               
2025-07-17T08:44:28.1553829Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1554151Z       Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1554426Z         x                        1024(0x400)                        
2025-07-17T08:44:28.1554706Z         y                        1024(0x400)                        
2025-07-17T08:44:28.1554971Z         z                        1024(0x400)                        
2025-07-17T08:44:28.1555275Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1555577Z       Grid Max Size per Dimension:
2025-07-17T08:44:28.1555829Z         x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1556111Z         y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1556391Z         z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1556708Z       FBarrier Max Size:       32                                 
2025-07-17T08:44:28.1556999Z *******                  
2025-07-17T08:44:28.1557212Z Agent 5                  
2025-07-17T08:44:28.1557419Z *******                  
2025-07-17T08:44:28.1557652Z   Name:                    gfx90a                             
2025-07-17T08:44:28.1557965Z   Uuid:                    GPU-1aa655a92f77fcda               
2025-07-17T08:44:28.1558311Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.1558650Z   Vendor Name:             AMD                                
2025-07-17T08:44:28.1558977Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:44:28.1559306Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:44:28.1559635Z   Float Round Mode:        NEAR                               
2025-07-17T08:44:28.1559972Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:44:28.1560303Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:44:28.1560625Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:44:28.1560949Z   Queue Type:              MULTI                              
2025-07-17T08:44:28.1561252Z   Node:                    4                                  
2025-07-17T08:44:28.1561559Z   Device Type:             GPU                                
2025-07-17T08:44:28.1561844Z   Cache Info:              
2025-07-17T08:44:28.1562084Z     L1:                      16(0x10) KB                        
2025-07-17T08:44:28.1562375Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:44:28.1562668Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:44:28.1562981Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:44:28.1563318Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:44:28.1563660Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:44:28.1563976Z   BDFID:                   4352                               
2025-07-17T08:44:28.1564294Z   Internal Node ID:        4                                  
2025-07-17T08:44:28.1564623Z   Compute Unit:            104                                
2025-07-17T08:44:28.1564939Z   SIMDs per CU:            4                                  
2025-07-17T08:44:28.1565259Z   Shader Engines:          8                                  
2025-07-17T08:44:28.1565595Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:44:28.1566074Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:44:28.1566437Z   Coherent Host Access:    FALSE                              
2025-07-17T08:44:28.1566741Z   Memory Properties:       
2025-07-17T08:44:28.1566983Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:44:28.1567291Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:44:28.1567757Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:44:28.1568101Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1568409Z   Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1568667Z     x                        1024(0x400)                        
2025-07-17T08:44:28.1568945Z     y                        1024(0x400)                        
2025-07-17T08:44:28.1569212Z     z                        1024(0x400)                        
2025-07-17T08:44:28.1569507Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:44:28.1569845Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:44:28.1570173Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1570466Z   Grid Max Size per Dimension:
2025-07-17T08:44:28.1570707Z     x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1570986Z     y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1571263Z     z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1571577Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:44:28.1571936Z   Packet Processor uCode:: 92                                 
2025-07-17T08:44:28.1572284Z   SDMA engine uCode::      9                                  
2025-07-17T08:44:28.1572621Z   IOMMU Support::          None                               
2025-07-17T08:44:28.1572911Z   Pool Info:               
2025-07-17T08:44:28.1573128Z     Pool 1                   
2025-07-17T08:44:28.1573405Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:44:28.1573739Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1574062Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1574400Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1574758Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1575122Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1575477Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1575770Z     Pool 2                   
2025-07-17T08:44:28.1576108Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:44:28.1576432Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1576741Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1577077Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1577420Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1577766Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1578102Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1578394Z     Pool 3                   
2025-07-17T08:44:28.1578653Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:44:28.1578976Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1579296Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1579637Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1579994Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1580341Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1580848Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1581155Z     Pool 4                   
2025-07-17T08:44:28.1581414Z       Segment:                 GROUP                              
2025-07-17T08:44:28.1581723Z       Size:                    64(0x40) KB                        
2025-07-17T08:44:28.1582042Z       Allocatable:             FALSE                              
2025-07-17T08:44:28.1582538Z       Alloc Granule:           0KB                                
2025-07-17T08:44:28.1582891Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:44:28.1583247Z       Alloc Alignment:         0KB                                
2025-07-17T08:44:28.1583594Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1583901Z   ISA Info:                
2025-07-17T08:44:28.1584128Z     ISA 1                    
2025-07-17T08:44:28.1584419Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:44:28.1584791Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:44:28.1585155Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:44:28.1585521Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1585891Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1586227Z       Fast f16:                TRUE                               
2025-07-17T08:44:28.1586561Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1586881Z       Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1587160Z         x                        1024(0x400)                        
2025-07-17T08:44:28.1587445Z         y                        1024(0x400)                        
2025-07-17T08:44:28.1587726Z         z                        1024(0x400)                        
2025-07-17T08:44:28.1588170Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1604144Z       Grid Max Size per Dimension:
2025-07-17T08:44:28.1604547Z         x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1604896Z         y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1605202Z         z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1605544Z       FBarrier Max Size:       32                                 
2025-07-17T08:44:28.1605865Z *******                  
2025-07-17T08:44:28.1606084Z Agent 6                  
2025-07-17T08:44:28.1606303Z *******                  
2025-07-17T08:44:28.1606553Z   Name:                    gfx90a                             
2025-07-17T08:44:28.1606881Z   Uuid:                    GPU-aaccf1cf8b4ae211               
2025-07-17T08:44:28.1607231Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.1607580Z   Vendor Name:             AMD                                
2025-07-17T08:44:28.1607917Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:44:28.1608247Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:44:28.1608584Z   Float Round Mode:        NEAR                               
2025-07-17T08:44:28.1608919Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:44:28.1609251Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:44:28.1609576Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:44:28.1609896Z   Queue Type:              MULTI                              
2025-07-17T08:44:28.1610190Z   Node:                    5                                  
2025-07-17T08:44:28.1610488Z   Device Type:             GPU                                
2025-07-17T08:44:28.1610778Z   Cache Info:              
2025-07-17T08:44:28.1611027Z     L1:                      16(0x10) KB                        
2025-07-17T08:44:28.1611504Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:44:28.1611830Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:44:28.1612167Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:44:28.1612501Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:44:28.1612837Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:44:28.1613308Z   BDFID:                   5120                               
2025-07-17T08:44:28.1613627Z   Internal Node ID:        5                                  
2025-07-17T08:44:28.1613955Z   Compute Unit:            104                                
2025-07-17T08:44:28.1614278Z   SIMDs per CU:            4                                  
2025-07-17T08:44:28.1614598Z   Shader Engines:          8                                  
2025-07-17T08:44:28.1614930Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:44:28.1615272Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:44:28.1615619Z   Coherent Host Access:    FALSE                              
2025-07-17T08:44:28.1616019Z   Memory Properties:       
2025-07-17T08:44:28.1616269Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:44:28.1616579Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:44:28.1616920Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:44:28.1617261Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1617566Z   Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1617831Z     x                        1024(0x400)                        
2025-07-17T08:44:28.1618103Z     y                        1024(0x400)                        
2025-07-17T08:44:28.1618365Z     z                        1024(0x400)                        
2025-07-17T08:44:28.1618659Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:44:28.1618995Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:44:28.1619324Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1619621Z   Grid Max Size per Dimension:
2025-07-17T08:44:28.1619862Z     x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1620133Z     y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1620405Z     z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1620717Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:44:28.1621089Z   Packet Processor uCode:: 92                                 
2025-07-17T08:44:28.1621437Z   SDMA engine uCode::      9                                  
2025-07-17T08:44:28.1621771Z   IOMMU Support::          None                               
2025-07-17T08:44:28.1622066Z   Pool Info:               
2025-07-17T08:44:28.1622284Z     Pool 1                   
2025-07-17T08:44:28.1622559Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:44:28.1622886Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1623211Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1623543Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1623897Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1624251Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1624589Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1624877Z     Pool 2                   
2025-07-17T08:44:28.1625145Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:44:28.1625470Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1625779Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1626110Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1626707Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1627063Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1627401Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1627692Z     Pool 3                   
2025-07-17T08:44:28.1628102Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:44:28.1628409Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1628716Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1629042Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1629384Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1629732Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1630068Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1630360Z     Pool 4                   
2025-07-17T08:44:28.1630611Z       Segment:                 GROUP                              
2025-07-17T08:44:28.1630910Z       Size:                    64(0x40) KB                        
2025-07-17T08:44:28.1631218Z       Allocatable:             FALSE                              
2025-07-17T08:44:28.1631543Z       Alloc Granule:           0KB                                
2025-07-17T08:44:28.1631891Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:44:28.1632230Z       Alloc Alignment:         0KB                                
2025-07-17T08:44:28.1632565Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1632856Z   ISA Info:                
2025-07-17T08:44:28.1633067Z     ISA 1                    
2025-07-17T08:44:28.1633343Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:44:28.1633707Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:44:28.1634052Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:44:28.1634393Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1634741Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1635063Z       Fast f16:                TRUE                               
2025-07-17T08:44:28.1635391Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1635707Z       Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1635984Z         x                        1024(0x400)                        
2025-07-17T08:44:28.1636255Z         y                        1024(0x400)                        
2025-07-17T08:44:28.1636525Z         z                        1024(0x400)                        
2025-07-17T08:44:28.1636823Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1637117Z       Grid Max Size per Dimension:
2025-07-17T08:44:28.1637366Z         x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1637637Z         y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1637906Z         z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1638211Z       FBarrier Max Size:       32                                 
2025-07-17T08:44:28.1638499Z *******                  
2025-07-17T08:44:28.1638704Z Agent 7                  
2025-07-17T08:44:28.1638898Z *******                  
2025-07-17T08:44:28.1639128Z   Name:                    gfx90a                             
2025-07-17T08:44:28.1639432Z   Uuid:                    GPU-27c44677f57701f4               
2025-07-17T08:44:28.1639762Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.1640090Z   Vendor Name:             AMD                                
2025-07-17T08:44:28.1640412Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:44:28.1640874Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:44:28.1641210Z   Float Round Mode:        NEAR                               
2025-07-17T08:44:28.1641551Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:44:28.1641874Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:44:28.1642319Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:44:28.1642628Z   Queue Type:              MULTI                              
2025-07-17T08:44:28.1642922Z   Node:                    6                                  
2025-07-17T08:44:28.1643228Z   Device Type:             GPU                                
2025-07-17T08:44:28.1643511Z   Cache Info:              
2025-07-17T08:44:28.1643746Z     L1:                      16(0x10) KB                        
2025-07-17T08:44:28.1644033Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:44:28.1644325Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:44:28.1644636Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:44:28.1644964Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:44:28.1645295Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:44:28.1645606Z   BDFID:                   44544                              
2025-07-17T08:44:28.1645920Z   Internal Node ID:        6                                  
2025-07-17T08:44:28.1646258Z   Compute Unit:            104                                
2025-07-17T08:44:28.1646576Z   SIMDs per CU:            4                                  
2025-07-17T08:44:28.1646896Z   Shader Engines:          8                                  
2025-07-17T08:44:28.1647220Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:44:28.1647556Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:44:28.1647905Z   Coherent Host Access:    FALSE                              
2025-07-17T08:44:28.1648208Z   Memory Properties:       
2025-07-17T08:44:28.1648450Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:44:28.1648748Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:44:28.1649084Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:44:28.1649427Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1649737Z   Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1649998Z     x                        1024(0x400)                        
2025-07-17T08:44:28.1650269Z     y                        1024(0x400)                        
2025-07-17T08:44:28.1650541Z     z                        1024(0x400)                        
2025-07-17T08:44:28.1650832Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:44:28.1651162Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:44:28.1651496Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1651794Z   Grid Max Size per Dimension:
2025-07-17T08:44:28.1652033Z     x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1652301Z     y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1652570Z     z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1652888Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:44:28.1653254Z   Packet Processor uCode:: 92                                 
2025-07-17T08:44:28.1653593Z   SDMA engine uCode::      9                                  
2025-07-17T08:44:28.1653927Z   IOMMU Support::          None                               
2025-07-17T08:44:28.1654214Z   Pool Info:               
2025-07-17T08:44:28.1654425Z     Pool 1                   
2025-07-17T08:44:28.1654696Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:44:28.1655157Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1655486Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1655816Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1656246Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1656597Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1657094Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1657379Z     Pool 2                   
2025-07-17T08:44:28.1657643Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:44:28.1657966Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1658273Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1658605Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1658950Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1659305Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1659642Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1659934Z     Pool 3                   
2025-07-17T08:44:28.1660198Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:44:28.1660509Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1660821Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1661148Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1661486Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1661837Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1662184Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1662473Z     Pool 4                   
2025-07-17T08:44:28.1662723Z       Segment:                 GROUP                              
2025-07-17T08:44:28.1663026Z       Size:                    64(0x40) KB                        
2025-07-17T08:44:28.1663332Z       Allocatable:             FALSE                              
2025-07-17T08:44:28.1663659Z       Alloc Granule:           0KB                                
2025-07-17T08:44:28.1664004Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:44:28.1664353Z       Alloc Alignment:         0KB                                
2025-07-17T08:44:28.1664687Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1664980Z   ISA Info:                
2025-07-17T08:44:28.1665192Z     ISA 1                    
2025-07-17T08:44:28.1665463Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:44:28.1665827Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:44:28.1666176Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:44:28.1666521Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1666873Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1667198Z       Fast f16:                TRUE                               
2025-07-17T08:44:28.1667522Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1667840Z       Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1668119Z         x                        1024(0x400)                        
2025-07-17T08:44:28.1668397Z         y                        1024(0x400)                        
2025-07-17T08:44:28.1668674Z         z                        1024(0x400)                        
2025-07-17T08:44:28.1668978Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1669276Z       Grid Max Size per Dimension:
2025-07-17T08:44:28.1669686Z         x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1669968Z         y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1670234Z         z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1670539Z       FBarrier Max Size:       32                                 
2025-07-17T08:44:28.1670828Z *******                  
2025-07-17T08:44:28.1671160Z Agent 8                  
2025-07-17T08:44:28.1671362Z *******                  
2025-07-17T08:44:28.1671594Z   Name:                    gfx90a                             
2025-07-17T08:44:28.1671899Z   Uuid:                    GPU-e97bf651691dab9b               
2025-07-17T08:44:28.1672226Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.1672559Z   Vendor Name:             AMD                                
2025-07-17T08:44:28.1672881Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:44:28.1673202Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:44:28.1673531Z   Float Round Mode:        NEAR                               
2025-07-17T08:44:28.1673862Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:44:28.1674184Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:44:28.1674503Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:44:28.1674820Z   Queue Type:              MULTI                              
2025-07-17T08:44:28.1675121Z   Node:                    7                                  
2025-07-17T08:44:28.1675420Z   Device Type:             GPU                                
2025-07-17T08:44:28.1675709Z   Cache Info:              
2025-07-17T08:44:28.1675944Z     L1:                      16(0x10) KB                        
2025-07-17T08:44:28.1676222Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:44:28.1676505Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:44:28.1676818Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:44:28.1677143Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:44:28.1677468Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:44:28.1677769Z   BDFID:                   45824                              
2025-07-17T08:44:28.1678069Z   Internal Node ID:        7                                  
2025-07-17T08:44:28.1678395Z   Compute Unit:            104                                
2025-07-17T08:44:28.1678709Z   SIMDs per CU:            4                                  
2025-07-17T08:44:28.1679028Z   Shader Engines:          8                                  
2025-07-17T08:44:28.1679358Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:44:28.1679695Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:44:28.1680038Z   Coherent Host Access:    FALSE                              
2025-07-17T08:44:28.1680337Z   Memory Properties:       
2025-07-17T08:44:28.1680578Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:44:28.1680886Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:44:28.1681217Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:44:28.1681549Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1681856Z   Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1682110Z     x                        1024(0x400)                        
2025-07-17T08:44:28.1682372Z     y                        1024(0x400)                        
2025-07-17T08:44:28.1682632Z     z                        1024(0x400)                        
2025-07-17T08:44:28.1682921Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:44:28.1683248Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:44:28.1683576Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1684022Z   Grid Max Size per Dimension:
2025-07-17T08:44:28.1684271Z     x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1684545Z     y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1684812Z     z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1685126Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:44:28.1685614Z   Packet Processor uCode:: 92                                 
2025-07-17T08:44:28.1685952Z   SDMA engine uCode::      9                                  
2025-07-17T08:44:28.1686283Z   IOMMU Support::          None                               
2025-07-17T08:44:28.1686564Z   Pool Info:               
2025-07-17T08:44:28.1686779Z     Pool 1                   
2025-07-17T08:44:28.1687047Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:44:28.1687374Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1687693Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1688026Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1688374Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1688723Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1689059Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1689351Z     Pool 2                   
2025-07-17T08:44:28.1689617Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:44:28.1689936Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1690249Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1690572Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1690914Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1691267Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1691605Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1691893Z     Pool 3                   
2025-07-17T08:44:28.1692150Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:44:28.1692462Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1692775Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1693101Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1693441Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1693785Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1694122Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1694410Z     Pool 4                   
2025-07-17T08:44:28.1694661Z       Segment:                 GROUP                              
2025-07-17T08:44:28.1694960Z       Size:                    64(0x40) KB                        
2025-07-17T08:44:28.1695273Z       Allocatable:             FALSE                              
2025-07-17T08:44:28.1695597Z       Alloc Granule:           0KB                                
2025-07-17T08:44:28.1695997Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:44:28.1696347Z       Alloc Alignment:         0KB                                
2025-07-17T08:44:28.1696679Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1696968Z   ISA Info:                
2025-07-17T08:44:28.1697178Z     ISA 1                    
2025-07-17T08:44:28.1697449Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:44:28.1697806Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:44:28.1698149Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:44:28.1698653Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1699011Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1699332Z       Fast f16:                TRUE                               
2025-07-17T08:44:28.1699688Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1700142Z       Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1700412Z         x                        1024(0x400)                        
2025-07-17T08:44:28.1700687Z         y                        1024(0x400)                        
2025-07-17T08:44:28.1700956Z         z                        1024(0x400)                        
2025-07-17T08:44:28.1701252Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1701549Z       Grid Max Size per Dimension:
2025-07-17T08:44:28.1701798Z         x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1702074Z         y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1702166Z         z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1702296Z       FBarrier Max Size:       32                                 
2025-07-17T08:44:28.1702369Z *******                  
2025-07-17T08:44:28.1702446Z Agent 9                  
2025-07-17T08:44:28.1702519Z *******                  
2025-07-17T08:44:28.1702628Z   Name:                    gfx90a                             
2025-07-17T08:44:28.1702732Z   Uuid:                    GPU-272948773bb8e400               
2025-07-17T08:44:28.1702868Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.1702982Z   Vendor Name:             AMD                                
2025-07-17T08:44:28.1703101Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:44:28.1703214Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:44:28.1703342Z   Float Round Mode:        NEAR                               
2025-07-17T08:44:28.1703461Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:44:28.1703577Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:44:28.1703690Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:44:28.1703802Z   Queue Type:              MULTI                              
2025-07-17T08:44:28.1703904Z   Node:                    8                                  
2025-07-17T08:44:28.1704020Z   Device Type:             GPU                                
2025-07-17T08:44:28.1704101Z   Cache Info:              
2025-07-17T08:44:28.1704195Z     L1:                      16(0x10) KB                        
2025-07-17T08:44:28.1704290Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:44:28.1704391Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:44:28.1704513Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:44:28.1704638Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:44:28.1704761Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:44:28.1704860Z   BDFID:                   36352                              
2025-07-17T08:44:28.1704986Z   Internal Node ID:        8                                  
2025-07-17T08:44:28.1705105Z   Compute Unit:            104                                
2025-07-17T08:44:28.1705218Z   SIMDs per CU:            4                                  
2025-07-17T08:44:28.1705337Z   Shader Engines:          8                                  
2025-07-17T08:44:28.1705463Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:44:28.1705590Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:44:28.1705724Z   Coherent Host Access:    FALSE                              
2025-07-17T08:44:28.1705811Z   Memory Properties:       
2025-07-17T08:44:28.1705912Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:44:28.1706174Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:44:28.1706305Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:44:28.1706430Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1706527Z   Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1706739Z     x                        1024(0x400)                        
2025-07-17T08:44:28.1706829Z     y                        1024(0x400)                        
2025-07-17T08:44:28.1706919Z     z                        1024(0x400)                        
2025-07-17T08:44:28.1707038Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:44:28.1707169Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:44:28.1707285Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1707374Z   Grid Max Size per Dimension:
2025-07-17T08:44:28.1707467Z     x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1707561Z     y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1707651Z     z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1707787Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:44:28.1707926Z   Packet Processor uCode:: 92                                 
2025-07-17T08:44:28.1708055Z   SDMA engine uCode::      9                                  
2025-07-17T08:44:28.1708177Z   IOMMU Support::          None                               
2025-07-17T08:44:28.1708257Z   Pool Info:               
2025-07-17T08:44:28.1708332Z     Pool 1                   
2025-07-17T08:44:28.1708464Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:44:28.1708573Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1708705Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1708829Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1708977Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1709104Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1709238Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1709319Z     Pool 2                   
2025-07-17T08:44:28.1709450Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:44:28.1709553Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1709680Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1709800Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1709942Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1710067Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1710199Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1710281Z     Pool 3                   
2025-07-17T08:44:28.1710402Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:44:28.1710510Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1710632Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1710770Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1710906Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1711032Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1711166Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1711248Z     Pool 4                   
2025-07-17T08:44:28.1711359Z       Segment:                 GROUP                              
2025-07-17T08:44:28.1711593Z       Size:                    64(0x40) KB                        
2025-07-17T08:44:28.1711722Z       Allocatable:             FALSE                              
2025-07-17T08:44:28.1711850Z       Alloc Granule:           0KB                                
2025-07-17T08:44:28.1711990Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:44:28.1712119Z       Alloc Alignment:         0KB                                
2025-07-17T08:44:28.1712393Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1712475Z   ISA Info:                
2025-07-17T08:44:28.1712548Z     ISA 1                    
2025-07-17T08:44:28.1712693Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:44:28.1712826Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:44:28.1712954Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:44:28.1713090Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1713238Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1713344Z       Fast f16:                TRUE                               
2025-07-17T08:44:28.1713476Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1713575Z       Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1713671Z         x                        1024(0x400)                        
2025-07-17T08:44:28.1713761Z         y                        1024(0x400)                        
2025-07-17T08:44:28.1713847Z         z                        1024(0x400)                        
2025-07-17T08:44:28.1713970Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1714058Z       Grid Max Size per Dimension:
2025-07-17T08:44:28.1714157Z         x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1714252Z         y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1714349Z         z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1714474Z       FBarrier Max Size:       32                                 
2025-07-17T08:44:28.1714553Z *******                  
2025-07-17T08:44:28.1714627Z Agent 10                 
2025-07-17T08:44:28.1714701Z *******                  
2025-07-17T08:44:28.1714802Z   Name:                    gfx90a                             
2025-07-17T08:44:28.1714920Z   Uuid:                    GPU-f3cd2202cbe079af               
2025-07-17T08:44:28.1715050Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.1715169Z   Vendor Name:             AMD                                
2025-07-17T08:44:28.1715288Z   Feature:                 KERNEL_DISPATCH                    
2025-07-17T08:44:28.1715410Z   Profile:                 BASE_PROFILE                       
2025-07-17T08:44:28.1715537Z   Float Round Mode:        NEAR                               
2025-07-17T08:44:28.1715667Z   Max Queue Number:        128(0x80)                          
2025-07-17T08:44:28.1715780Z   Queue Min Size:          64(0x40)                           
2025-07-17T08:44:28.1715903Z   Queue Max Size:          131072(0x20000)                    
2025-07-17T08:44:28.1716016Z   Queue Type:              MULTI                              
2025-07-17T08:44:28.1716112Z   Node:                    9                                  
2025-07-17T08:44:28.1716235Z   Device Type:             GPU                                
2025-07-17T08:44:28.1716316Z   Cache Info:              
2025-07-17T08:44:28.1716418Z     L1:                      16(0x10) KB                        
2025-07-17T08:44:28.1716509Z     L2:                      8192(0x2000) KB                    
2025-07-17T08:44:28.1716619Z   Chip ID:                 29708(0x740c)                      
2025-07-17T08:44:28.1716736Z   ASIC Revision:           1(0x1)                             
2025-07-17T08:44:28.1716863Z   Cacheline Size:          128(0x80)                          
2025-07-17T08:44:28.1717124Z   Max Clock Freq. (MHz):   1700                               
2025-07-17T08:44:28.1717234Z   BDFID:                   37632                              
2025-07-17T08:44:28.1717356Z   Internal Node ID:        9                                  
2025-07-17T08:44:28.1717480Z   Compute Unit:            104                                
2025-07-17T08:44:28.1717717Z   SIMDs per CU:            4                                  
2025-07-17T08:44:28.1717844Z   Shader Engines:          8                                  
2025-07-17T08:44:28.1717967Z   Shader Arrs. per Eng.:   1                                  
2025-07-17T08:44:28.1718098Z   WatchPts on Addr. Ranges:4                                  
2025-07-17T08:44:28.1718228Z   Coherent Host Access:    FALSE                              
2025-07-17T08:44:28.1718319Z   Memory Properties:       
2025-07-17T08:44:28.1718413Z   Features:                KERNEL_DISPATCH 
2025-07-17T08:44:28.1718544Z   Fast F16 Operation:      TRUE                               
2025-07-17T08:44:28.1718664Z   Wavefront Size:          64(0x40)                           
2025-07-17T08:44:28.1718795Z   Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1718888Z   Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1718981Z     x                        1024(0x400)                        
2025-07-17T08:44:28.1719073Z     y                        1024(0x400)                        
2025-07-17T08:44:28.1719159Z     z                        1024(0x400)                        
2025-07-17T08:44:28.1719283Z   Max Waves Per CU:        32(0x20)                           
2025-07-17T08:44:28.1719407Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-17T08:44:28.1719527Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1719612Z   Grid Max Size per Dimension:
2025-07-17T08:44:28.1719711Z     x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1719807Z     y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1719901Z     z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1720032Z   Max fbarriers/Workgrp:   32                                 
2025-07-17T08:44:28.1720180Z   Packet Processor uCode:: 92                                 
2025-07-17T08:44:28.1720302Z   SDMA engine uCode::      9                                  
2025-07-17T08:44:28.1720435Z   IOMMU Support::          None                               
2025-07-17T08:44:28.1720513Z   Pool Info:               
2025-07-17T08:44:28.1720591Z     Pool 1                   
2025-07-17T08:44:28.1720718Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-17T08:44:28.1720828Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1720951Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1721075Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1721219Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1721344Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1721475Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1721554Z     Pool 2                   
2025-07-17T08:44:28.1721681Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-17T08:44:28.1721788Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1721909Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1722031Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1722167Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1722295Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1722425Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1722630Z     Pool 3                   
2025-07-17T08:44:28.1722763Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-17T08:44:28.1722865Z       Size:                    67092480(0x3ffc000) KB             
2025-07-17T08:44:28.1722996Z       Allocatable:             TRUE                               
2025-07-17T08:44:28.1723121Z       Alloc Granule:           4KB                                
2025-07-17T08:44:28.1723388Z       Alloc Recommended Granule:2048KB                             
2025-07-17T08:44:28.1723515Z       Alloc Alignment:         4KB                                
2025-07-17T08:44:28.1723646Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1723723Z     Pool 4                   
2025-07-17T08:44:28.1723840Z       Segment:                 GROUP                              
2025-07-17T08:44:28.1723940Z       Size:                    64(0x40) KB                        
2025-07-17T08:44:28.1724065Z       Allocatable:             FALSE                              
2025-07-17T08:44:28.1724184Z       Alloc Granule:           0KB                                
2025-07-17T08:44:28.1724326Z       Alloc Recommended Granule:0KB                                
2025-07-17T08:44:28.1724448Z       Alloc Alignment:         0KB                                
2025-07-17T08:44:28.1724581Z       Accessible by all:       FALSE                              
2025-07-17T08:44:28.1724664Z   ISA Info:                
2025-07-17T08:44:28.1724740Z     ISA 1                    
2025-07-17T08:44:28.1724879Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-17T08:44:28.1725015Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-17T08:44:28.1725137Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-17T08:44:28.1725278Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1725410Z       Default Rounding Mode:   NEAR                               
2025-07-17T08:44:28.1725525Z       Fast f16:                TRUE                               
2025-07-17T08:44:28.1725654Z       Workgroup Max Size:      1024(0x400)                        
2025-07-17T08:44:28.1725757Z       Workgroup Max Size per Dimension:
2025-07-17T08:44:28.1725856Z         x                        1024(0x400)                        
2025-07-17T08:44:28.1725952Z         y                        1024(0x400)                        
2025-07-17T08:44:28.1726046Z         z                        1024(0x400)                        
2025-07-17T08:44:28.1726163Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-17T08:44:28.1726256Z       Grid Max Size per Dimension:
2025-07-17T08:44:28.1726347Z         x                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1726445Z         y                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1726534Z         z                        4294967295(0xffffffff)             
2025-07-17T08:44:28.1726667Z       FBarrier Max Size:       32                                 
2025-07-17T08:44:28.1726741Z *** Done ***             
2025-07-17T08:44:28.1726835Z + rocminfo
2025-07-17T08:44:28.1726929Z + grep -E 'Name:.*\sgfx|Marketing'
2025-07-17T08:44:28.3941299Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-17T08:44:28.3942149Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-17T08:44:28.3942800Z   Name:                    gfx90a                             
2025-07-17T08:44:28.3943263Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.3943670Z   Name:                    gfx90a                             
2025-07-17T08:44:28.3944066Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.3944463Z   Name:                    gfx90a                             
2025-07-17T08:44:28.3944868Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.3945523Z   Name:                    gfx90a                             
2025-07-17T08:44:28.3945946Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.3946379Z   Name:                    gfx90a                             
2025-07-17T08:44:28.3946777Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.3947175Z   Name:                    gfx90a                             
2025-07-17T08:44:28.3947766Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.3948164Z   Name:                    gfx90a                             
2025-07-17T08:44:28.3948560Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.3948953Z   Name:                    gfx90a                             
2025-07-17T08:44:28.3949343Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-17T08:44:28.4116215Z + MAYBE_ROCM=rocm/
2025-07-17T08:44:28.4116962Z + [[ linux-jammy-rocm-py3.10 == *xpu* ]]
2025-07-17T08:44:28.4117479Z + [[ linux-jammy-rocm-py3.10 != *-bazel-* ]]
2025-07-17T08:44:28.4117958Z + pip_install ninja==1.10.2
2025-07-17T08:44:28.4118485Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-07-17T08:44:28.4119163Z + python3 -m pip install --progress-bar off ninja==1.10.2
2025-07-17T08:44:28.8333010Z Collecting ninja==1.10.2
2025-07-17T08:44:28.8584894Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2025-07-17T08:44:28.8667143Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2025-07-17T08:44:29.4309481Z Installing collected packages: ninja
2025-07-17T08:44:29.4310715Z   Attempting uninstall: ninja
2025-07-17T08:44:29.4317940Z     Found existing installation: ninja 1.11.1.3
2025-07-17T08:44:29.4340938Z     Uninstalling ninja-1.11.1.3:
2025-07-17T08:44:29.4432542Z       Successfully uninstalled ninja-1.11.1.3
2025-07-17T08:44:29.4791865Z Successfully installed ninja-1.10.2
2025-07-17T08:44:29.5512785Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-07-17T08:44:29.5515740Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-07-17T08:44:29.5517432Z + [[ linux-jammy-rocm-py3.10 == *aarch64* ]]
2025-07-17T08:44:29.5517976Z + [[ linux-jammy-rocm-py3.10 == *asan* ]]
2025-07-17T08:44:29.5518482Z + [[ linux-jammy-rocm-py3.10 == *-debug* ]]
2025-07-17T08:44:29.5518981Z + [[ linux-jammy-rocm-py3.10 != *-bazel-* ]]
2025-07-17T08:44:29.5519699Z + echo 'We are not in debug mode: linux-jammy-rocm-py3.10. Expect the assertion to pass'
2025-07-17T08:44:29.5520607Z We are not in debug mode: linux-jammy-rocm-py3.10. Expect the assertion to pass
2025-07-17T08:44:29.5521239Z + cd test
2025-07-17T08:44:29.5521766Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2025-07-17T08:44:31.3212900Z + [[ inductor == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2025-07-17T08:44:31.3213247Z + [[ inductor == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2025-07-17T08:44:31.3213578Z + [[ inductor == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]]
2025-07-17T08:44:31.3218660Z + DYNAMO_BENCHMARK_FLAGS=()
2025-07-17T08:44:31.3219115Z + [[ inductor == *pr_time_benchmarks* ]]
2025-07-17T08:44:31.3219488Z + [[ inductor == *dynamo_eager* ]]
2025-07-17T08:44:31.3219840Z + [[ inductor == *aot_eager* ]]
2025-07-17T08:44:31.3220157Z + [[ inductor == *aot_inductor* ]]
2025-07-17T08:44:31.3220494Z + [[ inductor == *max_autotune_inductor* ]]
2025-07-17T08:44:31.3220841Z + [[ inductor == *inductor* ]]
2025-07-17T08:44:31.3221152Z + [[ inductor != *perf* ]]
2025-07-17T08:44:31.3221475Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor)
2025-07-17T08:44:31.3221814Z + [[ inductor == *dynamic* ]]
2025-07-17T08:44:31.3222559Z + [[ inductor == *cpu* ]]
2025-07-17T08:44:31.3222895Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2025-07-17T08:44:31.3239570Z + [[ linux-jammy-rocm-py3.10 == *libtorch* ]]
2025-07-17T08:44:31.3239902Z + [[ linux-jammy-rocm-py3.10 == *-bazel-* ]]
2025-07-17T08:44:31.3245072Z + cd test
2025-07-17T08:44:31.3246477Z + python -c 'import torch; print(torch.__config__.show())'
2025-07-17T08:44:32.8119458Z PyTorch built with:
2025-07-17T08:44:32.8119907Z   - GCC 11.4
2025-07-17T08:44:32.8120266Z   - C++ Version: 201703
2025-07-17T08:44:32.8121126Z   - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-07-17T08:44:32.8122179Z   - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-07-17T08:44:32.8122868Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-07-17T08:44:32.8123390Z   - LAPACK is enabled (usually provided by MKL)
2025-07-17T08:44:32.8123966Z   - NNPACK is enabled
2025-07-17T08:44:32.8124482Z   - CPU capability usage: AVX2
2025-07-17T08:44:32.8124998Z   - HIP Runtime 6.4.43483
2025-07-17T08:44:32.8125449Z   - MIOpen 3.4.0
2025-07-17T08:44:32.8125857Z   - Magma 2.7.2
2025-07-17T08:44:32.8130697Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=a38f433be2e94a64b095a44ba39879d02d0c2316, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 
2025-07-17T08:44:32.8134761Z 
2025-07-17T08:44:33.0969554Z + cd test
2025-07-17T08:44:33.0970101Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2025-07-17T08:44:34.3209155Z ATen/Parallel:
2025-07-17T08:44:34.3209617Z 	at::get_num_threads() : 128
2025-07-17T08:44:34.3210063Z 	at::get_num_interop_threads() : 128
2025-07-17T08:44:34.3210539Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-07-17T08:44:34.3210979Z 	omp_get_max_threads() : 128
2025-07-17T08:44:34.3211808Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-07-17T08:44:34.3212836Z 	mkl_get_max_threads() : 128
2025-07-17T08:44:34.3213537Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-07-17T08:44:34.3214299Z std::thread::hardware_concurrency() : 128
2025-07-17T08:44:34.3214855Z Environment variables:
2025-07-17T08:44:34.3215301Z 	OMP_NUM_THREADS : [not set]
2025-07-17T08:44:34.3215767Z 	MKL_NUM_THREADS : [not set]
2025-07-17T08:44:34.3216229Z ATen parallel backend: OpenMP
2025-07-17T08:44:34.3216427Z 
2025-07-17T08:44:34.8414942Z + [[ inductor == *numpy_2* ]]
2025-07-17T08:44:34.8415501Z + [[ linux-jammy-rocm-py3.10 == *aarch64* ]]
2025-07-17T08:44:34.8416152Z + [[ inductor == *backward* ]]
2025-07-17T08:44:34.8416568Z + [[ inductor == *xla* ]]
2025-07-17T08:44:34.8416960Z + [[ inductor == *executorch* ]]
2025-07-17T08:44:34.8417384Z + [[ inductor == \j\i\t\_\l\e\g\a\c\y ]]
2025-07-17T08:44:34.8417868Z + [[ linux-jammy-rocm-py3.10 == *libtorch* ]]
2025-07-17T08:44:34.8418345Z + [[ inductor == distributed ]]
2025-07-17T08:44:34.8418761Z + [[ inductor == *operator_benchmark* ]]
2025-07-17T08:44:34.8419762Z + [[ inductor == *inductor_distributed* ]]
2025-07-17T08:44:34.8420281Z + [[ inductor == *inductor-halide* ]]
2025-07-17T08:44:34.8420758Z + [[ inductor == *inductor-triton-cpu* ]]
2025-07-17T08:44:34.8421287Z + [[ inductor == *inductor-micro-benchmark* ]]
2025-07-17T08:44:34.8421794Z + [[ inductor == *huggingface* ]]
2025-07-17T08:44:34.8422389Z + [[ inductor == *timm* ]]
2025-07-17T08:44:34.8423404Z + [[ inductor == cachebench ]]
2025-07-17T08:44:34.8424100Z + [[ inductor == verify_cachebench ]]
2025-07-17T08:44:34.8424844Z + [[ inductor == *torchbench* ]]
2025-07-17T08:44:34.8425580Z + [[ inductor == *inductor_cpp_wrapper* ]]
2025-07-17T08:44:34.8426199Z + [[ inductor == *inductor* ]]
2025-07-17T08:44:34.8426575Z + install_torchvision
2025-07-17T08:44:34.8426872Z + local orig_preload
2025-07-17T08:44:34.8427153Z + local commit
2025-07-17T08:44:34.8427431Z ++ get_pinned_commit vision
2025-07-17T08:44:34.8427775Z ++ cat .github/ci_commit_pins/vision.txt
2025-07-17T08:44:34.8438340Z + commit=966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-07-17T08:44:34.8438695Z + orig_preload=
2025-07-17T08:44:34.8438934Z + '[' -n '' ']'
2025-07-17T08:44:34.8439486Z + pip_install --no-use-pep517 git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-07-17T08:44:34.8440185Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-07-17T08:44:34.8440976Z + python3 -m pip install --progress-bar off --no-use-pep517 git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-07-17T08:44:35.1937654Z Collecting git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-07-17T08:44:35.1944708Z   Cloning https://github.com/pytorch/vision.git (to revision 966da7e46f65d6d49df3e31214470a4fe5cc8e66) to /tmp/pip-req-build-v8cg_cys
2025-07-17T08:44:35.1984579Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-v8cg_cys
2025-07-17T08:44:37.5551672Z   Running command git rev-parse -q --verify 'sha^966da7e46f65d6d49df3e31214470a4fe5cc8e66'
2025-07-17T08:44:37.5589833Z   Running command git fetch -q https://github.com/pytorch/vision.git 966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-07-17T08:44:37.9336734Z   Running command git checkout -q 966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-07-17T08:44:38.4394818Z   Resolved https://github.com/pytorch/vision.git to commit 966da7e46f65d6d49df3e31214470a4fe5cc8e66
2025-07-17T08:44:41.1857587Z   Preparing metadata (setup.py) ... [?25l- \ | / - done
2025-07-17T08:44:41.1897939Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.22.0a0+966da7e) (1.22.4)
2025-07-17T08:44:41.1901883Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.22.0a0+966da7e) (2.9.0a0+gita38f433)
2025-07-17T08:44:41.1907169Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.22.0a0+966da7e) (11.0.0)
2025-07-17T08:44:41.1969866Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.22.0a0+966da7e) (3.18.0)
2025-07-17T08:44:41.1974747Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.22.0a0+966da7e) (4.14.1)
2025-07-17T08:44:41.1980095Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.22.0a0+966da7e) (1.13.3)
2025-07-17T08:44:41.1983426Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.22.0a0+966da7e) (2.8.8)
2025-07-17T08:44:41.1987252Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.22.0a0+966da7e) (3.1.6)
2025-07-17T08:44:41.1991053Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.22.0a0+966da7e) (2025.5.1)
2025-07-17T08:44:41.2005165Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch->torchvision==0.22.0a0+966da7e) (1.3.0)
2025-07-17T08:44:41.2481211Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.22.0a0+966da7e) (3.0.2)
2025-07-17T08:44:41.2528342Z Building wheels for collected packages: torchvision
2025-07-17T08:44:41.2614634Z   DEPRECATION: Building 'torchvision' using the legacy setup.py bdist_wheel mechanism, which will be removed in a future version. pip 25.3 will enforce this behaviour change. A possible replacement is to use the standardized build interface by setting the `--use-pep517` option, (possibly combined with `--no-build-isolation`), or adding a `pyproject.toml` file to the source tree of 'torchvision'. Discussion can be found at https://github.com/pypa/pip/issues/6334
2025-07-17T08:45:40.1177343Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done
2025-07-17T08:45:40.1207064Z [?25h  Created wheel for torchvision: filename=torchvision-0.22.0a0+966da7e-cp310-cp310-linux_x86_64.whl size=1570326 sha256=dd095e7f67908c5695d103d8ec4a836fbdcb1bc21a3b460ded75f6e689c1eb55
2025-07-17T08:45:40.1210961Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/9c/9d/3e/42fa2d5ac6ba44a90363f8fff0fa9e712e24d4f977637c81cb
2025-07-17T08:45:40.1245918Z Successfully built torchvision
2025-07-17T08:45:40.6316558Z Installing collected packages: torchvision
2025-07-17T08:45:41.0525618Z Successfully installed torchvision-0.22.0a0+966da7e
2025-07-17T08:45:41.2059823Z + '[' -n '' ']'
2025-07-17T08:45:41.2060217Z + test_inductor_shard 1
2025-07-17T08:45:41.2060606Z + [[ -z 2 ]]
2025-07-17T08:45:41.2060997Z + python tools/dynamo/verify_dynamo.py
2025-07-17T08:45:43.2129610Z Python version: 3.10.18
2025-07-17T08:45:43.2130162Z `torch` version: 2.9.0a0+gita38f433
2025-07-17T08:45:43.2130646Z CUDA version: None
2025-07-17T08:45:43.2130992Z ROCM version: 6.4
2025-07-17T08:45:43.2131210Z 
2025-07-17T08:45:55.0704831Z All required checks passed
2025-07-17T08:45:56.4829926Z + python test/run_test.py --inductor --include test_modules test_ops test_ops_gradients test_torch --shard 1 2 --verbose
2025-07-17T08:45:59.4543143Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T08:45:59.4545306Z   import pkg_resources
2025-07-17T08:46:00.0964132Z No TD results found
2025-07-17T08:46:00.0966184Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T08:46:00.2955193Z Ignoring disabled issues:  ['']
2025-07-17T08:46:00.3208687Z Found test times from artifacts
2025-07-17T08:46:00.4242125Z Found test times from artifacts
2025-07-17T08:46:00.4283721Z Running all tests
2025-07-17T08:46:00.4292117Z Running parallel tests on 8 processes
2025-07-17T08:46:00.4293662Z Name: tests to run (est. time: 92.28min)
2025-07-17T08:46:00.4293965Z   Serial tests (4):
2025-07-17T08:46:00.4294203Z     test_torch 1/4
2025-07-17T08:46:00.4294416Z     test_torch 2/4
2025-07-17T08:46:00.4294627Z     test_torch 3/4
2025-07-17T08:46:00.4294836Z     test_torch 4/4
2025-07-17T08:46:00.4295053Z   Parallel tests (12):
2025-07-17T08:46:00.4295289Z     test_modules 8/8
2025-07-17T08:46:00.4295506Z     test_ops 3/20
2025-07-17T08:46:00.4295717Z     test_ops 4/20
2025-07-17T08:46:00.4296026Z     test_ops 7/20
2025-07-17T08:46:00.4296236Z     test_ops 8/20
2025-07-17T08:46:00.4296444Z     test_ops 11/20
2025-07-17T08:46:00.4297021Z     test_ops 12/20
2025-07-17T08:46:00.4297240Z     test_ops 15/20
2025-07-17T08:46:00.4297447Z     test_ops 16/20
2025-07-17T08:46:00.4297652Z     test_ops 19/20
2025-07-17T08:46:00.4297851Z     test_ops 20/20
2025-07-17T08:46:00.4298075Z     test_ops_gradients 3/3
2025-07-17T08:46:00.4298335Z Name: excluded (est. time: 0.0min)
2025-07-17T08:46:00.4298814Z   Serial tests (0):
2025-07-17T08:46:00.4299036Z   Parallel tests (0):
2025-07-17T08:46:00.4299327Z Running test_torch 1/4 ... [2025-07-17 08:46:00.429559]
2025-07-17T08:46:00.4299669Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T08:46:00.4300833Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_torch.py', '--shard-id=1', '--num-shards=4', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 08:46:00.429928]
2025-07-17T08:53:45.3080831Z 
2025-07-17T08:53:45.3081639Z test_torch 1/4 was successful, full logs can be found in artifacts with path test/test-reports/test_torch_1.4_56a0673231b540cf_.log
2025-07-17T08:53:45.3161500Z Running 266 items in this shard: test/test_torch.py::TestBasicVitalSigns::test_dataloader_vitals, test/test_torch.py::TestTorch::test_Size, test/test_torch.py::TestTorch::test_backward_hooks_traverse, test/test_torch.py::TestTorch::test_batch_norm_cpu_inference, test/test_torch.py::TestTorch::test_copy_dtypes, test/test_torch.py::TestTorch::test_copy_float16, test/test_torch.py::TestTorch::test_cummax_neg_dim, test/test_torch.py::TestTorch::test_deterministic_flag, test/test_torch.py::TestTorch::test_dim_order, test/test_torch.py::TestTorch::test_dot_data_use, test/test_torch.py::TestTorch::test_empty_storage_view, test/test_torch.py::TestTorch::test_equal, test/test_torch.py::TestTorch::test_fill_diagonal, test/test_torch.py::TestTorch::test_format_scalar_meta, test/test_torch.py::TestTorch::test_index_add_all_dtypes, test/test_torch.py::TestTorch::test_index_fill_neg_dim, test/test_torch.py::TestTorch::test_is_nonzero, test/test_torch.py::TestTorch::test_is_same_size, test/test_torch.py::TestTorch::test_iter, test/test_torch.py::TestTorch::test_memory_format_contiguous_returns_same_tensor_if_already_satisfies, test/test_torch.py::TestTorch::test_nanmedian_neg_dim, test/test_torch.py::TestTorch::test_narrow_neg_dim, test/test_torch.py::TestTorch::test_newaxis_numpy_comparison, test/test_torch.py::TestTorch::test_newindex, test/test_torch.py::TestTorch::test_no_cuda_monkeypatch, test/test_torch.py::TestTorch::test_norm_neg_dim, test/test_torch.py::TestTorch::test_parallel_info, test/test_torch.py::TestTorch::test_pickle, test/test_torch.py::TestTorch::test_pin_memory, test/test_torch.py::TestTorch::test_reversed, test/test_torch.py::TestTorch::test_scatter_neg_dim, test/test_torch.py::TestTorch::test_show_config, test/test_torch.py::TestTorch::test_slice, test/test_torch.py::TestTorch::test_sobolengine_continuing, test/test_torch.py::TestTorch::test_sobolengine_draw_scrambled, test/test_torch.py::TestTorch::test_sobolengine_first_point, test/test_torch.py::TestTorch::test_storage_base_init, test/test_torch.py::TestTorch::test_storage_cycle_via_slots, test/test_torch.py::TestTorch::test_storage_dealloc_resurrected, test/test_torch.py::TestTorch::test_storage_dealloc_subclass_zombie, test/test_torch.py::TestTorch::test_storage_error_no_attribute, test/test_torch.py::TestTorch::test_storage_resurrected_weak_ref, test/test_torch.py::TestTorch::test_swap_basic, test/test_torch.py::TestTorch::test_swap_fail_slots, test/test_torch.py::TestTorch::test_tensor_cycle_via_dict, test/test_torch.py::TestTorch::test_tensor_slot_dealloc, test/test_torch.py::TestTorch::test_torch_from_file, test/test_torch.py::TestTorch::test_unflatten, test/test_torch.py::TestTorch::test_unsqueeze_neg_dim, test/test_torch.py::TestTorch::test_upsample_nearest1d_meta, test/test_torch.py::TestTorch::test_upsample_nearest2d_meta, test/test_torch.py::TestTorch::test_var_neg_dim, test/test_torch.py::TestVitalSignsCudaCUDA::test_cuda_vitals_gpu_only_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcdiv_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_False_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_True_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_True_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_True_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_True_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_self_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_self_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_self_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_self_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_self_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_addcmul_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_copy_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_fmod_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_map2_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_map_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_max_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_remainder_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_grad_p_lt_1_no_nan_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_norm_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_same_inputs_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_check_tensor_all_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_conv_transposed_backward_agnostic_to_memory_format_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy_math_view_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy_mem_overlap_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_corrcoef_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cpp_warnings_have_python_context_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cublas_config_nondeterministic_alert_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cummax_cummin_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deepcopy_scalar_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_uint32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_interpolate_bilinear_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_replication_pad2d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_dim_function_empty_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_exponential_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_geometric_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_geometric_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_geometric_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_geometric_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_geometric_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_accumulation_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_autocast_foreach0_fused0_AdamW_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_autocast_foreach0_fused0_Adam_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_autocast_foreach2_fused_True_Adam_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_unscale_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_update_scale_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gradient_extreme_cases_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gradient_extreme_cases_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gradient_spacing_list_length_error_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_hook_remove_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_deterministic_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amax_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amax_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amax_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_mean_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_mean_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_mean_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_prod_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_float8_e4m3fnuz, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_float8_e5m2fnuz, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_float8_e4m3fnuz, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_large_cumprod_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_log_normal_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_bool_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_complex128_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_complex64_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_float16_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_float64_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_inplace_noncontiguous_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_cpu_and_cuda_ops_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_propagation_rules_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_cpu_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_deterministic_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_empty_w_replacement_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_no_nondeterministic_alert_interpolate_bilinear_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_AdaptiveAvgPool2d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_MaxUnpool1d_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_MaxUnpool2d_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_MaxUnpool3d_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_grid_sample_2d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_histc_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_resize_quantized_cuda_quint2x4, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_resize_quantized_cuda_quint4x2, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_resize_quantized_cuda_quint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_reduced_type_float_copy_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_repeat_interleave_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scalar_check_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_add_bool_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_multiply_unsupported_dtypes_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_shift_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_uint64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_qint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_shape_empty_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_int8, test/test_torch.py::TestDevicePrecisionCUDA::test_advancedindex_mixed_cpu_devices_cuda, test/test_torch.py::TestDevicePrecisionCUDA::test_advancedindex_mixed_devices_error_cuda, test/test_torch.py::TestDevicePrecisionCUDA::test_copy_noncontig_cuda, test/test_torch.py::TestDevicePrecisionCUDA::test_cuda_device_idx_cuda, test/test_torch.py::TestDevicePrecisionCUDA::test_from_sequence_cuda_float16, test/test_torch.py::TestDevicePrecisionCUDA::test_from_sequence_cuda_int32, test/test_torch.py::TestDevicePrecisionCUDA::test_from_sequence_cuda_int64, test/test_torch.py::TestDevicePrecisionCUDA::test_from_sequence_cuda_int8, test/test_torch.py::TestDevicePrecisionCUDA::test_from_sequence_cuda_uint8, test/test_torch.py::TestDevicePrecisionCUDA::test_type_conversions_same_device_cuda
2025-07-17T08:53:45.3226313Z 
2025-07-17T08:53:45.3226547Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T08:53:45.3226967Z Uploading artifacts took 0.00 seconds
2025-07-17T08:53:45.3227278Z Running test_torch 2/4 ... [2025-07-17 08:53:45.312869]
2025-07-17T08:53:45.3227594Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T08:53:45.3228423Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_torch.py', '--shard-id=2', '--num-shards=4', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 08:53:45.313219]
2025-07-17T09:04:09.4997077Z 
2025-07-17T09:04:09.4998050Z test_torch 2/4 was successful, full logs can be found in artifacts with path test/test-reports/test_torch_2.4_98ae5be456969284_.log
2025-07-17T09:04:09.5073899Z Running 267 items in this shard: test/test_torch.py::TestBasicVitalSigns::test_basic_vitals, test/test_torch.py::TestTorch::test_RNGState, test/test_torch.py::TestTorch::test_Size_iter, test/test_torch.py::TestTorch::test_add_meta_scalar, test/test_torch.py::TestTorch::test_assert_async, test/test_torch.py::TestTorch::test_boxMullerState, test/test_torch.py::TestTorch::test_cat_neg_dim, test/test_torch.py::TestTorch::test_conj_neg_tolist, test/test_torch.py::TestTorch::test_copy_broadcast, test/test_torch.py::TestTorch::test_copy_many_to_one, test/test_torch.py::TestTorch::test_data_ptr_of_empty_view_with_storage, test/test_torch.py::TestTorch::test_deepcopy_parameter, test/test_torch.py::TestTorch::test_deterministic_fill_uninitialized_memory, test/test_torch.py::TestTorch::test_device, test/test_torch.py::TestTorch::test_element_size, test/test_torch.py::TestTorch::test_empty_meta, test/test_torch.py::TestTorch::test_from_buffer, test/test_torch.py::TestTorch::test_has_storage, test/test_torch.py::TestTorch::test_kthvalue_neg_dim, test/test_torch.py::TestTorch::test_linspace_logspace, test/test_torch.py::TestTorch::test_logcumsumexp_neg_dim, test/test_torch.py::TestTorch::test_memory_format, test/test_torch.py::TestTorch::test_parsing_intlist, test/test_torch.py::TestTorch::test_permute, test/test_torch.py::TestTorch::test_pickle_parameter, test/test_torch.py::TestTorch::test_renorm_neg_dim, test/test_torch.py::TestTorch::test_sobolengine_distribution_scrambled, test/test_torch.py::TestTorch::test_sobolengine_draw_base2_scrambled, test/test_torch.py::TestTorch::test_sobolengine_fast_forward, test/test_torch.py::TestTorch::test_sobolengine_high_dim, test/test_torch.py::TestTorch::test_sobolengine_raise, test/test_torch.py::TestTorch::test_squeeze_neg_dim, test/test_torch.py::TestTorch::test_storage_base_new, test/test_torch.py::TestTorch::test_storage_cycle_via_dict, test/test_torch.py::TestTorch::test_storage_fix_weakref_no_leak, test/test_torch.py::TestTorch::test_storage_from_tensor_dealloc_resurrected, test/test_torch.py::TestTorch::test_storage_preserve_nonhermetic_in_hermetic_context, test/test_torch.py::TestTorch::test_subclass_tensors, test/test_torch.py::TestTorch::test_sum_neg_dim, test/test_torch.py::TestTorch::test_t_not_2d_error, test/test_torch.py::TestTorch::test_tensor_base_init, test/test_torch.py::TestTorch::test_tensor_base_new, test/test_torch.py::TestTorch::test_tensor_cycle_via_slots, test/test_torch.py::TestTorch::test_tensor_dict_dealloc, test/test_torch.py::TestTorch::test_tensor_item_no_warning, test/test_torch.py::TestTorch::test_tensor_set, test/test_torch.py::TestTorch::test_tensor_set_errors, test/test_torch.py::TestTorch::test_tensor_with_grad_to_scalar_warning, test/test_torch.py::TestTorch::test_type_alias, test/test_torch.py::TestTorch::test_type_conversion_via_dtype_name, test/test_torch.py::TestTorch::test_unbind_neg_dim, test/test_torch.py::TestTorchDeviceTypeCUDA::test__local_scalar_dense_with_empty_tensor_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcdiv_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcdiv_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcdiv_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcdiv_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_cuda_errors_with_cpu_scalars_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_False_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_False_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_False_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_False_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_True_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_assertRaisesRegex_ignore_msg_non_native_device_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_p_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_self_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bfloat16_neg_abs_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_add_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_addcdiv_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_atan2_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_dist_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_lt_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_masked_select_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_min_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_uint64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_kstest_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_euclidean_large_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_large_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_check_tensor_internal_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_complex_half_experimental_warning_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_conv_transposed_large_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy_transpose_math_view_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_corrcoef_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cummin_discontiguous_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cumprod_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cumsum_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deepcopy_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deepcopy_scalar_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_complex32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_uint16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_uint64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_device_guard_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_errors_index_copy_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_exponential_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_exponential_kstest_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_exponential_kstest_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gather_backward_deterministic_path_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_geometric_kstest_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scale_will_not_overflow_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_clipping_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_clipping_separate_unscale_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gradient_all_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_add_deterministic_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_add_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amax_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amax_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amax_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amin_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amin_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_mean_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_mean_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_mean_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_prod_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_prod_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_prod_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_float8_e4m3fn, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_float8_e5m2, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_is_set_to_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_complex32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_uint64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_large_cumsum_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_logcumsumexp_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lognormal_kstest_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_bfloat16_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_complex64_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_float32_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_float64_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_int32_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_int8_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_int8_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_uint8_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_bool_tensor_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_discontiguous_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_preserved_after_permute_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_to_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_type_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_type_shortcuts_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_cpu_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_cpu_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_deterministic_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_deterministic_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_gpu_device_constrain_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_rng_state_advance_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_narrow_copy_non_contiguous_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_narrow_empty_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_no_nondeterministic_alert_interpolate_trilinear_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_AvgPool3d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_MaxUnpool2d_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_MaxUnpool3d_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_median_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_put_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_normal_kstest_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_normal_kstest_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nullary_op_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_params_invalidated_with_grads_invalidated_and_graph_partition_SGD_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_pdist_empty_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_pickle_gradscaler_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_pin_memory_from_constructor_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_add_non_unique_index_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_add_one_dim_deterministic_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_add_to_large_input_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_zero_size_index_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_serialization_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_default_tensor_type_warnings_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_skip_xla_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_qint32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_sync_warning_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_uint64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_uniform_kstest_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_uniform_kstest_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_warn_always_caught_cuda, test/test_torch.py::TestDevicePrecisionCUDA::test_clamp_cuda_float64, test/test_torch.py::TestDevicePrecisionCUDA::test_clamp_cuda_int64, test/test_torch.py::TestDevicePrecisionCUDA::test_from_sequence_cuda_int16, test/test_torch.py::TestDevicePrecisionCUDA::test_multidevice_serialization_cuda
2025-07-17T09:04:09.5138451Z 
2025-07-17T09:04:09.5138596Z Running test_torch 3/4 ... [2025-07-17 09:04:09.503967]
2025-07-17T09:04:09.5138921Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:04:09.5139755Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_torch.py', '--shard-id=3', '--num-shards=4', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:04:09.504362]
2025-07-17T09:13:17.2335432Z 
2025-07-17T09:13:17.2336584Z test_torch 3/4 was successful, full logs can be found in artifacts with path test/test-reports/test_torch_3.4_2a32d6e5a2039c78_.log
2025-07-17T09:13:17.2427318Z Running 265 items in this shard: test/test_torch.py::TestBasicVitalSigns::test_basic_vitals_read_write, test/test_torch.py::TestTorch::test_RNG_after_pickle, test/test_torch.py::TestTorch::test_Size_concat_non_tuple_sequence, test/test_torch.py::TestTorch::test_Size_concat_wildcard, test/test_torch.py::TestTorch::test_Size_scalar, test/test_torch.py::TestTorch::test_allow_tensor_metadata_change, test/test_torch.py::TestTorch::test_bf16_supported_on_cpu, test/test_torch.py::TestTorch::test_chunk_neg_dim, test/test_torch.py::TestTorch::test_cxx_flags, test/test_torch.py::TestTorch::test_doc, test/test_torch.py::TestTorch::test_dtype_is_signed, test/test_torch.py::TestTorch::test_error_msg_type_translation, test/test_torch.py::TestTorch::test_gather_neg_dim, test/test_torch.py::TestTorch::test_generator_cpu, test/test_torch.py::TestTorch::test_index_add_cornercase, test/test_torch.py::TestTorch::test_index_add_correctness, test/test_torch.py::TestTorch::test_index_add_neg_dim, test/test_torch.py::TestTorch::test_index_copy_neg_dim, test/test_torch.py::TestTorch::test_index_select_neg_dim, test/test_torch.py::TestTorch::test_invalid_generator_raises, test/test_torch.py::TestTorch::test_manual_seed, test/test_torch.py::TestTorch::test_median_neg_dim, test/test_torch.py::TestTorch::test_memory_format_empty, test/test_torch.py::TestTorch::test_multinomial_invalid_probs, test/test_torch.py::TestTorch::test_nbytes, test/test_torch.py::TestTorch::test_ndim, test/test_torch.py::TestTorch::test_parsing_double, test/test_torch.py::TestTorch::test_pickle_function, test/test_torch.py::TestTorch::test_pickle_size, test/test_torch.py::TestTorch::test_print, test/test_torch.py::TestTorch::test_prod_neg_dim, test/test_torch.py::TestTorch::test_qengine, test/test_torch.py::TestTorch::test_resizable, test/test_torch.py::TestTorch::test_select_neg_dim, test/test_torch.py::TestTorch::test_setting_real_imag_to_a_number, test/test_torch.py::TestTorch::test_sobolengine_continuing_scrambled, test/test_torch.py::TestTorch::test_sobolengine_distribution, test/test_torch.py::TestTorch::test_sobolengine_draw, test/test_torch.py::TestTorch::test_sobolengine_reset_scrambled, test/test_torch.py::TestTorch::test_sort_neg_dim, test/test_torch.py::TestTorch::test_split_neg_dim, test/test_torch.py::TestTorch::test_split_with_sizes_copy_out, test/test_torch.py::TestTorch::test_storage_casts, test/test_torch.py::TestTorch::test_storage_dealloc_subclass_resurrected, test/test_torch.py::TestTorch::test_storage_finalizer_dealloc, test/test_torch.py::TestTorch::test_storage_weakref_dealloc, test/test_torch.py::TestTorch::test_structseq_repr, test/test_torch.py::TestTorch::test_subclass_preserved, test/test_torch.py::TestTorch::test_tensor_dead_weak_ref, test/test_torch.py::TestTorch::test_tensor_finalizer_dealloc, test/test_torch.py::TestTorch::test_tensor_resurrected_weak_ref, test/test_torch.py::TestTorch::test_tensor_weakref_dealloc, test/test_torch.py::TestTorch::test_tensoriterator_output_setup, test/test_torch.py::TestTorch::test_terminate_handler_on_crash, test/test_torch.py::TestTorch::test_to, test/test_torch.py::TestTorch::test_transpose_neg_dim, test/test_torch.py::TestTorch::test_type, test/test_torch.py::TestTorch::test_typed_storage_deprecation_warning, test/test_torch.py::TestTorch::test_typed_storage_internal_no_warning, test/test_torch.py::TestTorch::test_warn_types, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_False_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_False_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_False_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_False_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_False_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_True_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_True_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_True_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_True_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_edge_cases_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_edge_cases_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_p_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_p_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_self_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_gt_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_le_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_lerp_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_masked_fill_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_masked_scatter_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_ne_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_pow_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_sub_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_no_inf_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_no_inf_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_empty_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_clone_all_dtypes_and_devices_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_clone_not_memory_dense_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy_transpose_math_view_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy_transpose_math_view_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_corrcoef_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cov_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deepcopy_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_cumsum_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_uint64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_uint16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_discontiguous_out_cumsum_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_expected_failure_xla_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_exponential_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_exponential_no_zero_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gather_backward_one_dim_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaler_pass_itself_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_autocast_foreach0_fused0_SGD_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_autocast_foreach2_fused_True_AdamW_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_autocast_foreach2_fused_True_SGD_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_multiple_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_state_dict_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_unscale_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_unscale_sparse_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gradient_all_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gradient_extreme_cases_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gradient_spacing_list_length_error_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gradient_type_promotion_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_add_large_inputs_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_put_non_accumulate_deterministic_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amax_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amax_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amin_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amin_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_mean_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_mean_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_mean_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_prod_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_prod_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_is_signed_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_float8_e4m3fn, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_float8_e5m2, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_float8_e5m2fnuz, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_uint32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_log_normal_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_log_normal_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_bool_tensor_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_float16_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_int16_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_int32_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_int64_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_uint8_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_clone_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_consistency_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_empty_like_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_AdaptiveMaxPool2d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_EmbeddingBag_max_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_MaxUnpool1d_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_MaxUnpool1d_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_MaxUnpool2d_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_ReflectionPad3d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_ReplicationPad1d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_grid_sample_3d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_histc_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_interpolate_bilinear_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_interpolate_linear_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_interpolate_trilinear_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_resize_quantized_cuda_qint32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_normal_kstest_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_normal_kstest_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_pairwise_distance_empty_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_params_invalidated_with_grads_invalidated_and_graph_partition_AdamW_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_params_invalidated_with_grads_invalidated_between_unscale_and_step_AdamW_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_params_invalidated_with_grads_invalidated_between_unscale_and_step_Adam_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_reduced_type_float_copy_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_bool_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_to_large_input_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_all_devices_non_blocking_True_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_uint16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_quint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_use_count_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_empty_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_uint32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_storage_type_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_ternary_op_mem_overlap_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_uniform_kstest_cuda_float32, test/test_torch.py::TestDevicePrecisionCUDA::test_clamp_cuda_float32, test/test_torch.py::TestDevicePrecisionCUDA::test_copy_broadcast_cuda, test/test_torch.py::TestDevicePrecisionCUDA::test_from_sequence_cuda_float32
2025-07-17T09:13:17.2491411Z 
2025-07-17T09:13:17.2491560Z Running test_torch 4/4 ... [2025-07-17 09:13:17.238226]
2025-07-17T09:13:17.2491887Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:13:17.2492717Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_torch.py', '--shard-id=4', '--num-shards=4', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:13:17.238539]
2025-07-17T09:23:35.0170361Z 
2025-07-17T09:23:35.0171391Z test_torch 4/4 was successful, full logs can be found in artifacts with path test/test-reports/test_torch_4.4_ee0ee742c940cdbf_.log
2025-07-17T09:23:35.0264061Z Running 270 items in this shard: test/test_torch.py::TestTorch::test_RNGStateAliasing, test/test_torch.py::TestTorch::test_apply, test/test_torch.py::TestTorch::test_as_subclass, test/test_torch.py::TestTorch::test_bmm_multithreaded, test/test_torch.py::TestTorch::test_check, test/test_torch.py::TestTorch::test_conj_physical_meta_stride, test/test_torch.py::TestTorch::test_contains, test/test_torch.py::TestTorch::test_copy_transpose, test/test_torch.py::TestTorch::test_cuda_not_built, test/test_torch.py::TestTorch::test_cummin_neg_dim, test/test_torch.py::TestTorch::test_cumprod_neg_dim, test/test_torch.py::TestTorch::test_cumsum_neg_dim, test/test_torch.py::TestTorch::test_data_ptr_of_empty_tensor_with_storage, test/test_torch.py::TestTorch::test_deepcopy_gradient, test/test_torch.py::TestTorch::test_dir, test/test_torch.py::TestTorch::test_doc_template, test/test_torch.py::TestTorch::test_from_file, test/test_torch.py::TestTorch::test_get_cpu_capability, test/test_torch.py::TestTorch::test_has_internal_overlap, test/test_torch.py::TestTorch::test_index_add, test/test_torch.py::TestTorch::test_invalid_arg_error_handling, test/test_torch.py::TestTorch::test_map, test/test_torch.py::TestTorch::test_map2, test/test_torch.py::TestTorch::test_max_neg_dim, test/test_torch.py::TestTorch::test_mean_neg_dim, test/test_torch.py::TestTorch::test_min_neg_dim, test/test_torch.py::TestTorch::test_mode_neg_dim, test/test_torch.py::TestTorch::test_new, test/test_torch.py::TestTorch::test_normal_shape, test/test_torch.py::TestTorch::test_numel, test/test_torch.py::TestTorch::test_parsing_int64, test/test_torch.py::TestTorch::test_pickle_dtype, test/test_torch.py::TestTorch::test_pickle_generator, test/test_torch.py::TestTorch::test_pickle_parameter_no_requires_grad, test/test_torch.py::TestTorch::test_pyobj_preserved, test/test_torch.py::TestTorch::test_set_flush_denormal, test/test_torch.py::TestTorch::test_size_neg_dim, test/test_torch.py::TestTorch::test_size_stride, test/test_torch.py::TestTorch::test_sizeof, test/test_torch.py::TestTorch::test_slow_test, test/test_torch.py::TestTorch::test_sobolengine_bounds, test/test_torch.py::TestTorch::test_sobolengine_bounds_scrambled, test/test_torch.py::TestTorch::test_sobolengine_default_dtype, test/test_torch.py::TestTorch::test_sobolengine_draw_base2, test/test_torch.py::TestTorch::test_sobolengine_fast_forward_scrambled, test/test_torch.py::TestTorch::test_sobolengine_reset, test/test_torch.py::TestTorch::test_std_neg_dim, test/test_torch.py::TestTorch::test_storage_byteswap, test/test_torch.py::TestTorch::test_storage_dead_weak_ref, test/test_torch.py::TestTorch::test_storage_dealloc, test/test_torch.py::TestTorch::test_storage_dict_dealloc, test/test_torch.py::TestTorch::test_storage_error, test/test_torch.py::TestTorch::test_storage_from_tensor_dealloc, test/test_torch.py::TestTorch::test_storage_from_tensor_dealloc_zombie, test/test_torch.py::TestTorch::test_storage_slot_dealloc, test/test_torch.py::TestTorch::test_tensor_ctor_scalar, test/test_torch.py::TestTorch::test_tensor_fix_weakref_no_leak, test/test_torch.py::TestTorch::test_tensor_ressurecting_clear, test/test_torch.py::TestTorch::test_tensor_where_scalar, test/test_torch.py::TestTorch::test_to_with_tensor, test/test_torch.py::TestTorch::test_topk_neg_dim, test/test_torch.py::TestTorch::test_unfold_neg_dim, test/test_torch.py::TestTorch::test_wildcard_import, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcdiv_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcdiv_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcdiv_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcdiv_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcdiv_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_addcmul_use_cpu_scalar_True_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_edge_cases_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_self_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bernoulli_self_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bool_tensor_value_change_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_div_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_eq_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_ge_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_broadcast_fn_mul_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_uint16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_bytes_to_scalar_cuda_uint32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cauchy_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_cuda_backward_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_large_batch_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_non_contiguous_batch_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_non_contiguous_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cdist_norm_batch_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_clone_zero_stride_dim_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_constants_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_complex32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy__cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_copy_all_dtypes_and_devices_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cummax_discontiguous_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_cumsum_64bit_indexing_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_empty_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_deterministic_resize_cuda_uint32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_diff_noncontig_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_dist_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_dtypetensor_warnings_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_exponential_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_exponential_kstest_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_exponential_kstest_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_exponential_no_zero_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_geometric_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_geometric_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_geometric_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_geometric_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaler_deprecated_warning_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_autocast_foreach_True_fused1_AdamW_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_autocast_foreach_True_fused1_Adam_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_autocast_foreach_True_fused1_SGD_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_grad_scaling_penalty_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gradient_all_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_gradient_spacing_list_length_error_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_copy_scalars_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_fill_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_put_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amax_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amin_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amin_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amin_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amin_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_amin_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_prod_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_prod_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_reduce_reduce_prod_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_index_select_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_int64_upsample3d_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_invalid_shapes_grid_sampler_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_item_cuda_uint16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_binary_op_no_materialize_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_lazy_clone_view_materialize_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_log_normal_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_bfloat16_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_bool_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_complex128_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_float32_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_int16_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_cuda_int64_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_fill_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_scatter_large_tensor_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_masked_select_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_factory_like_functions_preserve_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_memory_format_operators_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_module_share_memory_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_device_constrain_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_multinomial_empty_wo_replacement_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_AdaptiveAvgPool3d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_CTCLoss_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_FractionalMaxPool2d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_FractionalMaxPool3d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_MaxPool3d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_MaxUnpool3d_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_NLLLoss_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_ReflectionPad1d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_ReplicationPad2d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_ReplicationPad3d_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_bincount_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_interpolate_bicubic_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_kthvalue_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_alert_put_accumulate_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_nondeterministic_resize_quantized_cuda_qint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_parallel_cow_materialize_error_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_params_invalidated_with_grads_invalidated_and_graph_partition_Adam_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_params_invalidated_with_grads_invalidated_between_unscale_and_step_SGD_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_pdist_norm_large_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_accumulate_cuda_int32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_empty_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_put_mem_overlap_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_multiply_unsupported_dtypes_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_non_unique_index_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_operations_to_large_input_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_float64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_scatter_reduce_scalar_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_set_storage_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_all_devices_non_blocking_False_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_complex64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_cuda_uint32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_errors_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_bfloat16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_from_tensor_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_meta_ok_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_int64, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_quint4x2, test/test_torch.py::TestTorchDeviceTypeCUDA::test_storage_setitem_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_strides_propagation_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_bool, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_take_cuda_uint8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_complex128, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_int8, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_from_storage_cuda_uint16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_set_errors_multigpu_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_tensor_type_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_float32, test/test_torch.py::TestTorchDeviceTypeCUDA::test_typed_storage_meta_cuda_int16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_uniform_kstest_cuda_float16, test/test_torch.py::TestTorchDeviceTypeCUDA::test_untyped_storage_meta_cuda, test/test_torch.py::TestTorchDeviceTypeCUDA::test_where_scalar_handcrafted_values_cuda, test/test_torch.py::TestDevicePrecisionCUDA::test_device_serialization_cuda, test/test_torch.py::TestDevicePrecisionCUDA::test_from_sequence_cuda_float64, test/test_torch.py::TestDevicePrecisionCUDA::test_index_add_bfloat16_cuda
2025-07-17T09:23:35.0328951Z 
2025-07-17T09:23:35.0329183Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T09:23:35.0329603Z Uploading artifacts took 0.00 seconds
2025-07-17T09:23:35.0329916Z Running test_modules 8/8 ... [2025-07-17 09:23:35.024849]
2025-07-17T09:23:35.0330253Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:23:35.0331129Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_modules.py', '-m', 'serial', '--shard-id=8', '--num-shards=8', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:23:35.025150]
2025-07-17T09:23:41.1476977Z 
2025-07-17T09:23:41.1477717Z test_modules 8/8 was successful, full logs can be found in artifacts with path test/test-reports/test_modules_8.8_daa1c3c620b550a1_.log
2025-07-17T09:23:41.1478967Z Running 0 items in this shard:
2025-07-17T09:23:41.1479341Z 
2025-07-17T09:23:41.1480673Z Running test_ops 3/20 ... [2025-07-17 09:23:41.147701]
2025-07-17T09:23:41.1481427Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:23:41.1483154Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'serial', '--shard-id=3', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:23:41.148029]
2025-07-17T09:23:55.7315530Z 
2025-07-17T09:23:55.7316659Z test_ops 3/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_3.20_d3155365c758e6d9_.log
2025-07-17T09:23:55.7317636Z Running 0 items in this shard:
2025-07-17T09:23:55.7317914Z 
2025-07-17T09:23:55.7318154Z Running test_ops 4/20 ... [2025-07-17 09:23:55.731483]
2025-07-17T09:23:55.7318694Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:23:55.7320232Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'serial', '--shard-id=4', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:23:55.731791]
2025-07-17T09:24:11.5669987Z 
2025-07-17T09:24:11.5670591Z test_ops 4/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_4.20_da7faeb5107bf230_.log
2025-07-17T09:24:11.5671307Z Running 0 items in this shard:
2025-07-17T09:24:11.5671514Z 
2025-07-17T09:24:11.5671681Z Running test_ops 7/20 ... [2025-07-17 09:24:11.566919]
2025-07-17T09:24:11.5672081Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:24:11.5674380Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'serial', '--shard-id=7', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:24:11.567223]
2025-07-17T09:24:26.1509035Z 
2025-07-17T09:24:26.1509858Z test_ops 7/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_7.20_0ee406cb1ca43917_.log
2025-07-17T09:24:26.1510650Z Running 0 items in this shard:
2025-07-17T09:24:26.1510978Z 
2025-07-17T09:24:26.1511366Z Running test_ops 8/20 ... [2025-07-17 09:24:26.150808]
2025-07-17T09:24:26.1512032Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:24:26.1513597Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'serial', '--shard-id=8', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:24:26.151105]
2025-07-17T09:24:40.6346137Z 
2025-07-17T09:24:40.6346999Z test_ops 8/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_8.20_a11d12b9bee8c225_.log
2025-07-17T09:24:40.6348149Z Running 0 items in this shard:
2025-07-17T09:24:40.6348490Z 
2025-07-17T09:24:40.6348753Z Running test_ops 11/20 ... [2025-07-17 09:24:40.634623]
2025-07-17T09:24:40.6349252Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:24:40.6352255Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'serial', '--shard-id=11', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:24:40.635033]
2025-07-17T09:24:55.0182684Z 
2025-07-17T09:24:55.0183728Z test_ops 11/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_11.20_a72c4c1ed3b1f788_.log
2025-07-17T09:24:55.0184724Z Running 0 items in this shard:
2025-07-17T09:24:55.0185004Z 
2025-07-17T09:24:55.0185244Z Running test_ops 12/20 ... [2025-07-17 09:24:55.018218]
2025-07-17T09:24:55.0185787Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:24:55.0188701Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'serial', '--shard-id=12', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:24:55.018531]
2025-07-17T09:25:10.8036373Z 
2025-07-17T09:25:10.8037313Z test_ops 12/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_12.20_e4ae8614316ea2d6_.log
2025-07-17T09:25:10.8038634Z Running 0 items in this shard:
2025-07-17T09:25:10.8039024Z 
2025-07-17T09:25:10.8040024Z Running test_ops 15/20 ... [2025-07-17 09:25:10.803612]
2025-07-17T09:25:10.8040777Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:25:10.8042431Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'serial', '--shard-id=15', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:25:10.803926]
2025-07-17T09:25:25.1874938Z 
2025-07-17T09:25:25.1875613Z test_ops 15/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_15.20_91f922dd8509ec3c_.log
2025-07-17T09:25:25.1876228Z Running 0 items in this shard:
2025-07-17T09:25:25.1876394Z 
2025-07-17T09:25:25.1883197Z Running test_ops 16/20 ... [2025-07-17 09:25:25.187422]
2025-07-17T09:25:25.1883559Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:25:25.1884516Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'serial', '--shard-id=16', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:25:25.187738]
2025-07-17T09:25:39.6205498Z 
2025-07-17T09:25:39.6206442Z test_ops 16/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_16.20_c17fa84240194b92_.log
2025-07-17T09:25:39.6207417Z Running 0 items in this shard:
2025-07-17T09:25:39.6207697Z 
2025-07-17T09:25:39.6208314Z Running test_ops 19/20 ... [2025-07-17 09:25:39.620599]
2025-07-17T09:25:39.6208861Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:25:39.6211922Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'serial', '--shard-id=19', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:25:39.620923]
2025-07-17T09:25:54.1546342Z 
2025-07-17T09:25:54.1547303Z test_ops 19/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_19.20_c35cff8be542bc3a_.log
2025-07-17T09:25:54.1548439Z Running 0 items in this shard:
2025-07-17T09:25:54.1548761Z 
2025-07-17T09:25:54.1549040Z Running test_ops 20/20 ... [2025-07-17 09:25:54.154635]
2025-07-17T09:25:54.1549660Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:25:54.1554462Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'serial', '--shard-id=20', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:25:54.155160]
2025-07-17T09:26:08.5386827Z 
2025-07-17T09:26:08.5387486Z test_ops 20/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_20.20_604166f5b6b3e79a_.log
2025-07-17T09:26:08.5388061Z Running 0 items in this shard:
2025-07-17T09:26:08.5388222Z 
2025-07-17T09:26:08.5388397Z Running test_ops_gradients 3/3 ... [2025-07-17 09:26:08.538584]
2025-07-17T09:26:08.5388767Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:26:08.5391024Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops_gradients.py', '-m', 'serial', '--shard-id=3', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:26:08.538897]
2025-07-17T09:26:15.0618592Z 
2025-07-17T09:26:15.0619666Z test_ops_gradients 3/3 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_gradients_3.3_374dd19a6a5593bd_.log
2025-07-17T09:26:15.0620737Z Running 0 items in this shard:
2025-07-17T09:26:15.0621781Z 
2025-07-17T09:26:18.5883302Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:26:18.5886250Z   import pkg_resources
2025-07-17T09:26:18.7425612Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:26:18.7427103Z   import pkg_resources
2025-07-17T09:26:18.7860315Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:26:18.7862322Z   import pkg_resources
2025-07-17T09:26:18.7964634Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:26:18.7966589Z   import pkg_resources
2025-07-17T09:26:18.7969998Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:26:18.7971962Z   import pkg_resources
2025-07-17T09:26:18.8012892Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:26:18.8014883Z   import pkg_resources
2025-07-17T09:26:18.8109655Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:26:18.8111687Z   import pkg_resources
2025-07-17T09:26:18.8404669Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:26:18.8406895Z   import pkg_resources
2025-07-17T09:26:18.9881247Z Running test_modules 8/8 ... [2025-07-17 09:26:18.987902]
2025-07-17T09:26:18.9881676Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:26:18.9884876Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_modules.py', '-m', 'not serial', '--shard-id=8', '--num-shards=8', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:26:18.988283]
2025-07-17T09:26:19.2275390Z Running test_ops 3/20 ... [2025-07-17 09:26:19.227376]
2025-07-17T09:26:19.2275790Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:26:19.2279766Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'not serial', '--shard-id=3', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:26:19.227744]
2025-07-17T09:26:19.3002762Z Running test_ops 4/20 ... [2025-07-17 09:26:19.300114]
2025-07-17T09:26:19.3003169Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:26:19.3007210Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'not serial', '--shard-id=4', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:26:19.300526]
2025-07-17T09:26:19.3313645Z Running test_ops 7/20 ... [2025-07-17 09:26:19.331217]
2025-07-17T09:26:19.3314071Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:26:19.3318240Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'not serial', '--shard-id=7', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:26:19.331586]
2025-07-17T09:26:19.3436757Z Running test_ops 8/20 ... [2025-07-17 09:26:19.343532]
2025-07-17T09:26:19.3437164Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:26:19.3437549Z Running test_ops 11/20 ... [2025-07-17 09:26:19.343577]
2025-07-17T09:26:19.3437949Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:26:19.3441001Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'not serial', '--shard-id=8', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:26:19.343911]
2025-07-17T09:26:19.3442830Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'not serial', '--shard-id=11', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:26:19.343971]
2025-07-17T09:26:19.3443943Z Running test_ops 12/20 ... [2025-07-17 09:26:19.343979]
2025-07-17T09:26:19.3444333Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:26:19.3445801Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'not serial', '--shard-id=12', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:26:19.344401]
2025-07-17T09:26:19.3559439Z Running test_ops 15/20 ... [2025-07-17 09:26:19.355770]
2025-07-17T09:26:19.3559853Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:26:19.3563768Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'not serial', '--shard-id=15', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:26:19.356182]
2025-07-17T09:35:00.5273381Z 
2025-07-17T09:35:00.5274469Z test_modules 8/8 was successful, full logs can be found in artifacts with path test/test-reports/test_modules_8.8_af2a4b72c9a07e0a_.log
2025-07-17T09:35:00.5402938Z Running 427 items in this shard: test/test_modules.py::TestModuleCUDA::test_check_inplace_nn_Threshold_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_CircularPad1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_Conv3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_ConvTranspose1d_cuda_complex64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_ConvTranspose2d_cuda_complex128, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_ConvTranspose3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_CrossEntropyLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_FractionalMaxPool2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_GELU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_GRUCell_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_GroupNorm_cuda_float16, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_Hardtanh_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_Hardtanh_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_HuberLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_InstanceNorm1d_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_InstanceNorm2d_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_L1Loss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_LSTMCell_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_LayerNorm_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_LazyConv2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_LazyConv2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_LazyConvTranspose3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_LeakyReLU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_LogSigmoid_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_MSELoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_MarginRankingLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_MarginRankingLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_ReLU6_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_ReLU6_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_ReplicationPad1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_SELU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_Threshold_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_AdaptiveAvgPool1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_AdaptiveAvgPool2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_AdaptiveAvgPool3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_AvgPool2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_Bilinear_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_ConvTranspose1d_cuda_complex64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_ConvTranspose2d_cuda_complex128, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_CosineEmbeddingLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_CrossEntropyLoss_cuda_float16, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_CrossEntropyLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_GELU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_GRU_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_HuberLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_InstanceNorm3d_eval_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_LazyConv2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_LazyConv3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_LazyConv3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_LeakyReLU_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_LogSigmoid_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_LogSoftmax_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_MaxPool1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_MaxPool3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_Mish_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_MultiLabelSoftMarginLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_MultiMarginLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_RNN_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_ReflectionPad3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_ReplicationPad1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_ReplicationPad3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_SELU_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_SiLU_cuda_float32, test/test_modules.py::TestModuleCUDA::test_device_ctx_init_nn_ZeroPad2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_errors_nn_GRU_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_errors_nn_LSTM_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_errors_nn_LSTM_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_AdaptiveAvgPool2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_AdaptiveAvgPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_AvgPool2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_AvgPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_CircularPad1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_ConstantPad1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_ConstantPad2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_ConvTranspose1d_cuda_complex32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_ConvTranspose1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_ConvTranspose3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_CrossEntropyLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_ELU_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_GRU_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_GRU_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_Hardswish_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_HingeEmbeddingLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_InstanceNorm2d_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_KLDivLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_LazyConv2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_LazyConvTranspose2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_LogSigmoid_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_MaxPool1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_MaxPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_MultiLabelMarginLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_MultiMarginLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_PoissonNLLLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_Sigmoid_cuda_float32, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_SoftMarginLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_Tanhshrink_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_TransformerEncoderLayer_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_factory_kwargs_nn_ZeroPad1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_AdaptiveAvgPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_AdaptiveMaxPool3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_BatchNorm1d_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_BatchNorm1d_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_BatchNorm3d_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_CircularPad2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_CircularPad3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_ConstantPad2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_ConstantPad3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_Conv2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_Conv3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_CosineEmbeddingLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_Embedding_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_HingeEmbeddingLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_L1Loss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_LPPool1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_LSTMCell_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_LazyConvTranspose1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_LocalResponseNorm_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_MultiheadAttention_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_NLLLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_ReLU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_ReflectionPad1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_Softmax2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_Softmax_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_Softshrink_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_Threshold_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_Threshold_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_TransformerDecoderLayer_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_TransformerEncoder_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_Transformer_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_Transformer_cuda_float64, test/test_modules.py::TestModuleCUDA::test_forward_nn_ZeroPad1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_forward_nn_ZeroPad2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_grad_nn_BatchNorm1d_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_BatchNorm3d_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_CELU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_CTCLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_Embedding_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_GroupNorm_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_InstanceNorm3d_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_MultiheadAttention_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_PReLU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_RNNCell_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_ReplicationPad2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_grad_nn_Tanhshrink_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_BCEWithLogitsLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_CircularPad3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_ConvTranspose2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_LSTM_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_LayerNorm_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_Linear_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_LogSoftmax_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_MultiheadAttention_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_Sigmoid_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_Softmax_cuda_float64, test/test_modules.py::TestModuleCUDA::test_gradgrad_nn_TransformerEncoderLayer_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_AdaptiveMaxPool1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_BatchNorm3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_CircularPad2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_ConstantPad3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_Conv2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_ConvTranspose1d_cuda_complex128, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_ConvTranspose1d_cuda_complex32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_ConvTranspose2d_cuda_complex32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_CosineEmbeddingLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_GaussianNLLLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_GroupNorm_cuda_bfloat16, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_HingeEmbeddingLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_L1Loss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_MaxPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_Mish_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_MultiMarginLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_RMSNorm_cuda_float64, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_ReLU6_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_ReflectionPad1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_ReplicationPad3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_Sigmoid_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_SoftMarginLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_Softplus_cuda_float32, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_Softsign_cuda_float64, test/test_modules.py::TestModuleCUDA::test_if_train_and_eval_modes_differ_nn_Tanh_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_AdaptiveMaxPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_AvgPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_BCELoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_BCELoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_BatchNorm2d_eval_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_BatchNorm2d_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_CTCLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_Conv2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_FractionalMaxPool3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_GRUCell_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_Hardtanh_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_InstanceNorm3d_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_LazyConvTranspose3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_LocalResponseNorm_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_MSELoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_MaxPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_MaxPool3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_MaxPool3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_Mish_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_MultiMarginLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_MultiheadAttention_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_PReLU_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_PoissonNLLLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_RMSNorm_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_ReflectionPad3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_ReplicationPad1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_Softplus_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_Softshrink_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_Softshrink_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_Tanh_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_Tanhshrink_cuda_float64, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_Transformer_cuda_float32, test/test_modules.py::TestModuleCUDA::test_memory_format_nn_Transformer_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_AdaptiveAvgPool2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_AdaptiveAvgPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_AdaptiveMaxPool1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_AdaptiveMaxPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_BatchNorm1d_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_BatchNorm3d_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_CELU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_Conv1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_ConvTranspose1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_ConvTranspose2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_ConvTranspose3d_cuda_complex64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_Embedding_cuda_float32, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_GRUCell_cuda_float32, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_GRUCell_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_GRU_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_GroupNorm_cuda_float16, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_InstanceNorm3d_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_LSTMCell_cuda_float32, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_LSTM_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_LeakyReLU_cuda_float32, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_LogSigmoid_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_MultiLabelMarginLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_MultiLabelSoftMarginLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_SiLU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_Softmax2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_Softsign_cuda_float64, test/test_modules.py::TestModuleCUDA::test_multiple_device_transfer_nn_ZeroPad2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_AdaptiveAvgPool1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_AdaptiveAvgPool2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_AdaptiveMaxPool2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_AdaptiveMaxPool3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_AvgPool2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_BCELoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_BatchNorm1d_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_BatchNorm2d_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_BatchNorm2d_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_BatchNorm3d_eval_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_CELU_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_CELU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_ConvTranspose1d_cuda_complex32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_ConvTranspose1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_ConvTranspose2d_cuda_complex128, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_CosineEmbeddingLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_CosineEmbeddingLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_CrossEntropyLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_GLU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_GRU_eval_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_GRU_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_GRU_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_GroupNorm_cuda_float16, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_GroupNorm_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_Hardswish_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_Hardswish_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_HuberLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_InstanceNorm2d_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_InstanceNorm3d_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_LeakyReLU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_LogSigmoid_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_MaxPool2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_MultiMarginLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_MultiheadAttention_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_NLLLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_ReplicationPad2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_Sigmoid_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_Tanh_cuda_float32, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_ZeroPad1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_non_contiguous_tensors_nn_ZeroPad2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_AvgPool3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_BatchNorm2d_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_BatchNorm3d_eval_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_BatchNorm3d_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_CELU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_ConstantPad1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_ConstantPad3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_Conv2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_Conv2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_Conv3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_ConvTranspose1d_cuda_complex64, test/test_modules.py::TestModuleCUDA::test_repr_nn_ConvTranspose2d_cuda_complex32, test/test_modules.py::TestModuleCUDA::test_repr_nn_ConvTranspose2d_cuda_complex64, test/test_modules.py::TestModuleCUDA::test_repr_nn_Embedding_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_FractionalMaxPool2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_GELU_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_GRU_eval_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_GaussianNLLLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_GroupNorm_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_Hardshrink_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_InstanceNorm1d_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_InstanceNorm2d_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_InstanceNorm3d_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_LPPool1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_LSTM_train_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_LazyConv2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_LazyConvTranspose3d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_LocalResponseNorm_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_MaxPool1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_Mish_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_MultiLabelMarginLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_NLLLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_Softsign_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_Tanhshrink_cuda_float32, test/test_modules.py::TestModuleCUDA::test_repr_nn_Threshold_cuda_float64, test/test_modules.py::TestModuleCUDA::test_repr_nn_TransformerDecoderLayer_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_CircularPad3d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_ConstantPad2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_ConvTranspose1d_cuda_complex32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_ConvTranspose2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_ConvTranspose3d_cuda_complex32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_ELU_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_GLU_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_GRUCell_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_GRU_eval_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_HuberLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_LSTM_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_LSTM_train_mode_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_LazyConvTranspose2d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_LogSigmoid_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_MarginRankingLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_MultiMarginLoss_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_MultiheadAttention_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_NLLLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_PoissonNLLLoss_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_RNN_eval_mode_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_ReplicationPad2d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_Softplus_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_Threshold_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_TransformerDecoderLayer_cuda_float64, test/test_modules.py::TestModuleCUDA::test_save_load_nn_ZeroPad1d_cuda_float32, test/test_modules.py::TestModuleCUDA::test_save_load_nn_ZeroPad1d_cuda_float64, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_AdaptiveMaxPool1d_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_AvgPool1d_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_BCEWithLogitsLoss_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_BatchNorm3d_eval_mode_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_BatchNorm3d_train_mode_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_Bilinear_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_ConstantPad1d_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_CosineEmbeddingLoss_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_Embedding_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_GRU_eval_mode_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_GaussianNLLLoss_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_GaussianNLLLoss_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_HuberLoss_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_InstanceNorm3d_eval_mode_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_KLDivLoss_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_LPPool2d_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_LPPool2d_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_LSTM_train_mode_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_MarginRankingLoss_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_MultiLabelSoftMarginLoss_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_MultiMarginLoss_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_PReLU_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_PReLU_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_PoissonNLLLoss_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_RMSNorm_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_RNNCell_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_RNN_eval_mode_swap_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_Softmax_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_empty_nn_Transformer_swap_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_AdaptiveAvgPool3d_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_BCELoss_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_BCELoss_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_BatchNorm3d_eval_mode_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_BatchNorm3d_train_mode_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_BatchNorm3d_train_mode_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_CELU_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_CircularPad1d_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_CircularPad1d_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_CircularPad3d_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ConstantPad2d_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ConstantPad2d_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ConstantPad3d_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_Conv2d_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_Conv2d_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_CrossEntropyLoss_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_Hardswish_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_HingeEmbeddingLoss_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_InstanceNorm1d_eval_mode_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_InstanceNorm1d_eval_mode_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_InstanceNorm2d_eval_mode_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_InstanceNorm2d_eval_mode_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_LPPool3d_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_LSTMCell_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_LSTMCell_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_LayerNorm_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_LayerNorm_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_Linear_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_LocalResponseNorm_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_LogSoftmax_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_MarginRankingLoss_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_MaxPool1d_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_MaxPool2d_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_MaxPool3d_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_MaxPool3d_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_Mish_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_MultiheadAttention_eval_mode_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_NLLLoss_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_RMSNorm_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_RNN_train_mode_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ReLU6_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ReLU6_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ReLU_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ReflectionPad1d_swap_True_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ReplicationPad1d_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ReplicationPad2d_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_SiLU_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_Sigmoid_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_Softmax_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_Tanh_swap_False_set_grad_False_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_TransformerDecoderLayer_swap_False_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_TransformerEncoderLayer_eval_mode_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_TransformerEncoder_train_mode_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_Transformer_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ZeroPad2d_swap_True_set_grad_True_cuda_float32, test/test_modules.py::TestModuleCUDA::test_to_nn_ZeroPad3d_swap_False_set_grad_False_cuda_float32
2025-07-17T09:35:00.5530932Z 
2025-07-17T09:35:00.5531071Z Running test_ops 16/20 ... [2025-07-17 09:35:00.528281]
2025-07-17T09:35:00.5531388Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:35:00.5532238Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'not serial', '--shard-id=16', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:35:00.528601]
2025-07-17T09:37:22.2490414Z 
2025-07-17T09:37:22.2491441Z test_ops 12/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_12.20_d5ef068529ade2db_.log
2025-07-17T09:37:22.2958558Z Running 1565 items in this shard: test/test_ops.py::TestCommonCUDA::test_compare_cpu___rsub___cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_dsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_lerp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_new_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_transpose_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_var_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_addmm_decomposed_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_atleast_3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_cholesky_inverse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_inv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_log_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_ctc_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_max_unpool1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_normal_number_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_polar_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_reshape_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_resize_as__cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_rsub_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_unfold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_unsqueeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_view_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_chunk_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_empty_permuted_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_item_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_randn_like_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_sigmoid_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_squeeze_multiple_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes__refs__conversions_complex_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_cauchy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_copysign_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_diagonal_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_digamma_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_lcm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_logaddexp2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_lt_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_meshgrid_list_of_tensors_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_hinge_embedding_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_margin_ranking_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_mse_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_tanhshrink_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_positive_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_reshape_as_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_special_softmax_with_dtype_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_vsplit_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_amin_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_argmax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_atan2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_bitwise_and_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_bitwise_not_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cholesky_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_conj_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_corrcoef_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_double_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_erfc_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_exp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_flip_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_float_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_gather_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_histogramdd_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_index_reduce_amin_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_index_select_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_jiterator_2inputs_2outputs_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_le_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_diagonal_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_lstsq_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_log10_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_log_normal_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_argmin_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_neg_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_celu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_conv_transpose1d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_scaled_dot_product_attention_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_softmin_with_dtype_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_upsample_nearest_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_polygamma_polygamma_n_0_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_polygamma_polygamma_n_3_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_positive_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_prod_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_signal_windows_blackman_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_signbit_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_slice_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_sparse_mm_reduce_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_chebyshev_polynomial_t_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_squeeze_multiple_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_std_mean_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_unbind_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_var_unbiased_cuda, test/test_ops.py::TestCommonCUDA::test_errors_fft_fftn_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_adaptive_max_pool2d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_randn_like_layout2_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_zeros_like_layout3_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_hfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_ifftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_hstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_index_reduce_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_isposinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_maximum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_nanmean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_native_batch_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_pow_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_rsqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_trunc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_H_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_addcmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_addmv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_asinh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_broadcast_shapes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cov_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_diagflat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_irfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fliplr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_geometric_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_index_reduce_prod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_isclose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_kron_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_lcm_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_lerp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_cond_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_log10_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_log2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_log_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_log_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_logsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_sum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_max_binary_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_median_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_min_reduction_with_dim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_msort_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_ne_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_adaptive_max_pool1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_cosine_embedding_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_embedding_bag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_relu6_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_permute_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_permute_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_prod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_randint_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_randint_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_resize__cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_searchsorted_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_bessel_j1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_polygamma_special_polygamma_n_0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_scaled_modified_bessel_k1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_zeta_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_split_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sum_to_size_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_view_as_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_view_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_view_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_zeros_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_acosh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_amin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_bitwise_or_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_broadcast_tensors_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fft_ihfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fill_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_flipud_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_isposinf_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_linalg_diagonal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_logical_xor_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_maximum_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_mul_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_scatter_reduce_sum_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_select_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_chebyshev_polynomial_u_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_modified_bessel_k0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_unsqueeze_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples__unsafe_masked_index_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_acos_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addcdiv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addcmul_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addmm_decomposed_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_angle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cartesian_prod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cfloat_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cholesky_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_column_stack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diagonal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_div_floor_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_double_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_dsplit_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_dstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_hfftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ifft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ifft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_irfftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_flip_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_float_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_floor_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fmax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_gt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_put_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_reduce_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_reduce_mean_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_isnan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_ldl_factor_ex_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lu_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_matrix_rank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_svd_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_meshgrid_variadic_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mvlgamma_mvlgamma_p_3_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nansum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nextafter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_interpolate_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_linear_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pad_reflect_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_polygamma_polygamma_n_1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_pow_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_qr_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_rand_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_reciprocal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_resize_as__cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_resolve_neg_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_rot90_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_signal_windows_exponential_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_modified_bessel_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_shifted_chebyshev_polynomial_w_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_xlog1py_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sqrt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unfold_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unique_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unsqueeze_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unsqueeze_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_view_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_zeros_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_zeros_like_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_argwhere_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_broadcast_to_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_cat_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_ravel_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_signal_windows_gaussian_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_where_cuda_float64, test/test_ops.py::TestCommonCUDA::test_out_H_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out___rpow___cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_asin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_erfc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_hfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_ge_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_index_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_log10_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_log2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_normal__in_place_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_renorm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_view_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_acos_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cholesky_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_digamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_norm_subgradients_at_zero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_long_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_lt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_var_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_mm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_new_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_conv_transpose1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_conv_transpose3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_instance_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_linear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_put_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_addcdiv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_asinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_ceil_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cross_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_digamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_div_floor_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_irfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_index_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_pinv_hermitian_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_solve_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_svdvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_tensorsolve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_log_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_masked_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_normal_number_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_ormqr_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_rsqrt_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sigmoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_slice_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_special_i1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_special_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_take_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_select_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_i1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_modified_bessel_i1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_xlog1py_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_std_mean_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_tan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_transpose_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_unsqueeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning___getitem___cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_addcdiv_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_bucketize_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_clamp_max_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_conj_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_empty_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_fft2_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_frac_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_maximum_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_l1_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_roll_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_squeeze_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_to_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_true_divide_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__segment_reduce_offsets_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_addr_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cat_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_ceil_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_diagonal_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_empty_permuted_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_hfftn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_isinf_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_jiterator_binary_return_by_ref_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_matrix_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_log10_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_logical_or_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_mv_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_native_layer_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_conv_transpose1d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_leaky_relu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_max_unpool3d_grad_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_normalize_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_triplet_margin_with_distance_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_scatter_reduce_amax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_zeros_like_cuda, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_acos_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_asin_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_asinh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atan2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atanh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_logit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_rsqrt_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_legendre_polynomial_p_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_u_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_byte_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_char_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_char_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_double_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_short_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_abs_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_add_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addr_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amax_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amax_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_any_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_partial_views_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asin_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_2d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_left_shift_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_block_diag_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cat_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clone_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_column_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_column_stack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_column_stack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_physical_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cos_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_count_nonzero_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumsum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_embed_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_floor_rounding_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_no_rounding_mode_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_trunc_rounding_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_like_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eq_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_equal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erfinv_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expm1_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exponential_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfft_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fill_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flatten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_float_power_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_divide_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmax_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_gt_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hsplit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_i0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_igamma_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_add_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isfinite_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isinf_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isinf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isinf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isnan_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isneginf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isneginf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_item_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_item_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_le_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_cross_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_diagonal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_svdvals_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log10_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log1p_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logaddexp2_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_and_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_and_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_not_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_tensor_overload_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logsumexp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lt_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_maximum_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_variadic_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ne_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ne_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_neg_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_neg_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_ones_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_l1_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_log_softmax_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_margin_ranking_loss_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_mish_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_unshuffle_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_smooth_l1_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_positive_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ravel_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_real_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_roll_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_roll_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rot90_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsqrt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsqrt_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sigmoid_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sign_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_bessel_j1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i1_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i1_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i1e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_spherical_bessel_j0_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_xlog1py_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_zeta_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_std_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_std_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_take_along_dim_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trace_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_var_mean_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vsplit_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vstack_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_xlogy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs__conversions_polar_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_hsplit_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_hstack_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_narrow_copy_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_roll_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_trace_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bool_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_byte_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_chalf_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_chalf_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_char_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_double_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_half_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_short_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_abs_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcdiv_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcmul_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_all_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_all_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_any_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_partial_views_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_to_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_to_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cauchy_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clone_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_physical_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_contiguous_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_copysign_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_copysign_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_count_nonzero_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_deg2rad_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_copy_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_trunc_rounding_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dstack_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erf_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_copy_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expm1_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exponential_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft2_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftshift_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft2_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftn_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft2_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_rfftn_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flatten_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flipud_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_frexp_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_heaviside_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hstack_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_igammac_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_add_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_add_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isclose_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isnan_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isposinf_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_diagonal_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_matrix_norm_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_svdvals_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_vecdot_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log2_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_softmax_with_dtype_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_softmax_with_dtype_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_and_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_not_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logspace_tensor_overload_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lt_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_variadic_tensors_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_variadic_tensors_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mul_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ne_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_full_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_gelu_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hardtanh_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_log_softmax_with_dtype_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_margin_ranking_loss_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_unshuffle_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_poisson_nll_loss_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_smooth_l1_loss_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_triplet_margin_loss_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_normal__in_place_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ones_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_copy_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_pow_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_renorm_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rot90_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_round_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsqrt_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsub_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_signbit_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sin_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sin_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinc_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_softmax_with_dtype_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_bessel_j1_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_entr_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_erfcx_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_1_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtr_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtr_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_softmax_with_dtype_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_softmax_with_dtype_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_softmax_with_dtype_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_zeta_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_copy_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_copy_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_stack_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_std_mean_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_stft_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_copy_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tan_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tanh_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_indices_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_true_divide_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trunc_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_copy_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_copy_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_where_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_zeros_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bool_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_byte_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_byte_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cfloat_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_float_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_half_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_half_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_long_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_short_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_abs_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acosh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcdiv_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_alias_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_alias_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_all_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_any_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asinh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_2d_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_3d_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_tensors_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cat_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ceil_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ceil_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_chunk_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_chunk_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_min_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_physical_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_physical_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_physical_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cosh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_deg2rad_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_digamma_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dsplit_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_like_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_like_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eq_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_equal_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_equal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfc_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expm1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exponential_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftshift_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfftn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftshift_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfftn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fill_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flip_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_float_power_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_frac_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_heaviside_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hstack_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_i0_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_select_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isclose_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lerp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lgamma_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_cross_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_diagonal_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_tensor_overload_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_normal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_not_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_not_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_tensor_overload_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lt_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_masked_fill_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_masked_fill_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nan_to_num_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ne_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_full_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_ones_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_elu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_group_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_huber_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_log_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_margin_ranking_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_mse_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_shuffle_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_relu6_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmin_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_tanhshrink_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_threshold_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_triplet_margin_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_triplet_margin_loss_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_norm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_pow_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_pow_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rad2deg_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rad2deg_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_real_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_as_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_as_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rot90_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsqrt_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsqrt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsub_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sgn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sign_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sin_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinc_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_erfcx_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i0e_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_logit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_1_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtri_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_spherical_bessel_j0_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_split_with_sizes_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_square_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sub_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_take_along_dim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tan_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tanh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tanh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_triu_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_as_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vstack_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bfloat16_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bool_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cdouble_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_half_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_short_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amin_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_any_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_arange_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_1d_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_left_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_or_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_block_diag_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cat_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cat_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cauchy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_chunk_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_chunk_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_count_nonzero_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_deg2rad_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_embed_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_scatter_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_like_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfc_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expm1_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eye_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft2_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft2_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifftshift_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifftshift_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flatten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flatten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flatten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flip_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flip_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fliplr_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_float_power_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmin_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_frexp_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_i0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_i0_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_fill_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isclose_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isclose_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isnan_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isposinf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isreal_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_istft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_istft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_item_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_le_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_diagonal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_diagonal_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log10_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log1p_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logaddexp2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logaddexp_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_xor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logspace_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_mean_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_mul_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nan_to_num_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_full_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_full_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_zeros_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_zeros_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_alpha_dropout_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_channel_shuffle_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_channel_shuffle_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_dropout_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_dropout_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hardtanh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hardtanh_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_l1_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_margin_ranking_loss_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_unshuffle_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_prelu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_relu6_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmin_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_normal__in_place_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_pow_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_pow_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_prod_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_prod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_prod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rad2deg_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_randn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reciprocal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_renorm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_repeat_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_as_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_roll_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rot90_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsqrt_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsub_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsub_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_select_scatter_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_select_scatter_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sigmoid_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_bessel_j0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_erfcx_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1e_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_ndtr_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_ndtr_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_logit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtri_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtri_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtri_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_spherical_bessel_j0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_xlog1py_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_split_with_sizes_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sqrt_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_multiple_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_multiple_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tanh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_to_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tril_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tril_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_true_divide_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trunc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trunc_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_as_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_xlogy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_add_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_as_strided_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_asin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_atan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_atleast_2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cartesian_prod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_chalf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_clamp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_empty_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_fftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_hfft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_gather_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_inner_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_jiterator_4inputs_with_extra_args_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_eig_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_eigh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_eigvalsh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_householder_product_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_inv_ex_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_ldl_factor_ex_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_matrix_rank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_long_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_long_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_argmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_sum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_msort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_new_full_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nextafter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_glu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_outer_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_pinverse_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_reshape_as_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_signal_windows_kaiser_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_airy_ai_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_split_list_args_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_square_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sum_to_size_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_t_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_torch_ops_aten__efficient_attention_forward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_where_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_zeros_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_zeros_like_cuda_complex64, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_addmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_clone_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cov_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cumulative_trapezoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_diff_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fliplr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_lstsq_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_multi_dot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_lu_unpack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_min_binary_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_mm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nan_to_num_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nanmedian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_logsigmoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_norm_fro_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_permute_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_scatter_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_scatter_reduce_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_sign_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_torch_ops_aten__efficient_attention_forward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_unsafe_chunk_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_H_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_T_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input___getitem___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_chalf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_cross_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_cummax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_cumprod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_hfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_irfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_full_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_half_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_index_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_index_put_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_index_reduce_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_kron_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_le_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_lu_unpack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_masked_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_narrow_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_glu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_grid_sample_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_relu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_softsign_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_renorm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_repeat_interleave_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_sparse_mm_reduce_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_std_mean_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad___rmatmul___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_contiguous_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_cumprod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_expand_as_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_hfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_ge_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_lstsq_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_logaddexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_lt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_smooth_l1_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_put_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_ravel_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_signal_windows_blackman_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_signal_windows_general_cosine_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_bessel_j1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sum_to_size_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_tanh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_tensordot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator___rmod___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_atanh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_clamp_min_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_cos_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_diag_embed_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_dstack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_erfinv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_expand_as_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_index_fill_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_index_select_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_jiterator_binary_return_by_ref_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_tensorsolve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_logical_not_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_logical_xor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_mH_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_cumsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nanmean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_avg_pool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_avg_pool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_pairwise_distance_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_rms_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nonzero_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_real_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_reciprocal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_scatter_reduce_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_bessel_y0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_log_ndtr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_take_along_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_tile_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_to_sparse_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_trapezoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_trapz_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_unbind_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay___rpow___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_amax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_atan_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_broadcast_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_div_floor_rounding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_irfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_rfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_ldl_factor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_svdvals_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_maximum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_mv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_pairwise_distance_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_polygamma_polygamma_n_2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_sparse_mm_reduce_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_log_ndtr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_tanh_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view_T_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_addcmul_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_index_select_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_isnan_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_logical_not_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_logsumexp_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_narrow_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_new_full_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_prod_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_sqrt_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_square_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_sum_to_size_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_exp_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_fft_fft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_fill_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_index_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_item_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_jiterator_binary_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_pinv_singular_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_masked_prod_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_narrow_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_conv1d_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_silu_complex_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_to_sparse_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_vsplit_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_vstack_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view___rmatmul___cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_diagonal_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_lerp_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_logical_not_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_mul_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_new_empty_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_nn_functional_l1_loss_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_nn_functional_tanhshrink_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_positive_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_repeat_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_special_log_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_vdot_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_allclose_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_any_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_cdouble_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_clone_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_cos_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_empty_like_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_empty_permuted_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_eq_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_expand_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fft_irfftn_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_lu_factor_ex_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_svdvals_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_vector_norm_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_logical_xor_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_lu_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_lu_unpack_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_masked_logsumexp_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_pad_constant_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_silu_complex_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_sqrt_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_square_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_tanh_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_tile_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_unsqueeze_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view_H_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_T_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fft_fft_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fliplr_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_linalg_diagonal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nan_to_num_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nextafter_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_leaky_relu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_relu6_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_special_erfcx_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_zeros_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_any_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_atan2_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_bmm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_bool_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_broadcast_to_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cdouble_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_conj_physical_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_expand_as_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fft_fftn_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fft_irfftn_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fmin_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_gather_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_igammac_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_eigh_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_matrix_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_lu_unpack_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_amin_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_msort_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_new_empty_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_adaptive_avg_pool2d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_conv_transpose2d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_group_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_interpolate_bicubic_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_margin_ranking_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_multilabel_margin_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_relu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_pinverse_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_ravel_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_reciprocal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_scalar_tensor_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signal_windows_cosine_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signal_windows_general_hamming_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_hermite_polynomial_h_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_triu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_unfold_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_unique_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake_as_strided_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_T_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_as_strided_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_atleast_1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_bernoulli_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cdist_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cdouble_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_chunk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_empty_permuted_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_gcd_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_histc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_istft_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_cholesky_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_new_empty_strided_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_polar_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_randn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_roll_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_rsub_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_scatter_reduce_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_sinc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_std_mean_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_t_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_tanh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_tensor_split_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_combinations_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_bernoulli_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_chunk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_erfc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_flip_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_floor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fmod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_frac_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_grid_sampler_2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_index_put_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_log_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_movedim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_mv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_interpolate_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_relu6_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_ormqr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_polygamma_polygamma_n_2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_round_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_squeeze_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_t_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_tile_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_trapz_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_trunc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_unbind_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_unsqueeze_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_chalf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cholesky_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_corrcoef_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cov_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_digamma_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_hfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_ihfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_gather_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_diagonal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_narrow_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_native_batch_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_cosine_similarity_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_softsign_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_put_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_rsqrt_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_special_i1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_special_log_ndtr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_split_with_sizes_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_take_along_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_vsplit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_cumsum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_expand_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_eye_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_frexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_gather_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_imag_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_cond_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_eigvals_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_svd_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_argmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_matmul_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_narrow_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_native_layer_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_neg_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_relu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_randn_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_round_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_i0e_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_i1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_modified_bessel_i0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_true_divide_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_unravel_index_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_unsqueeze_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_arange_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_as_strided_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_atan2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_bitwise_or_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_cos_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_irfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_isposinf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_jiterator_unary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_lcm_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_ldexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_log1p_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_lt_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_mT_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_silu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_norm_fro_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_reshape_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_scalar_tensor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_sigmoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_sum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_tile_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_tensor_overload_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_ones_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_ones_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_ones_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_zeros_cuda_complex64, test/test_ops.py::TestTagsCUDA::test_tags__refs__conversions_long_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_acosh_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_amin_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_asin_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_atan2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_deg2rad_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_expand_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_hfft2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_ifft2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_float_power_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_ge_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_item_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_logical_xor_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_narrow_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_normal_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_renorm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_softmax_with_dtype_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_i1e_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_sub_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_var_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_atan_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_atleast_2d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_bitwise_left_shift_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags_diagonal_scatter_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_expand_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_geqrf_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_isnan_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_isneginf_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_logical_or_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_var_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_matmul_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_mode_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_new_empty_strided_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_ormqr_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_repeat_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_round_decimals_0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_signal_windows_exponential_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_signbit_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_ndtr_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_shifted_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_spherical_bessel_j0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_view_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_xlogy_cuda_float32
2025-07-17T09:37:22.3368356Z 
2025-07-17T09:37:22.3368498Z Running test_ops 19/20 ... [2025-07-17 09:37:22.262954]
2025-07-17T09:37:22.3368822Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:37:22.3369689Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'not serial', '--shard-id=19', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:37:22.263290]
2025-07-17T09:37:43.4309800Z 
2025-07-17T09:37:43.4310500Z test_ops 3/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_3.20_907f90f74e289b18_.log
2025-07-17T09:37:43.4833318Z Running 1729 items in this shard: test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs__conversions_double_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_block_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_empty_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_fmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_masked_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_new_empty_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_glu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_special_xlog1py_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__unsafe_masked_index_put_accumulate_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_as_strided_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_bitwise_left_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_compare_cpu_cdouble_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_gather_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_geometric_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_index_reduce_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_log_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_long_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_matmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_msort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_interpolate_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_put_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_select_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_sort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_special_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_T_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_atanh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_index_add_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_nn_functional_conv2d_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_select_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_stack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes___getitem___cuda, test/test_ops.py::TestCommonCUDA::test_dtypes___rmul___cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__native_batch_norm_legit_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs__conversions_half_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_add_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_atan2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_atleast_2d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_bitwise_left_shift_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_bitwise_right_shift_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_broadcast_shapes_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_bucketize_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fmin_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fmod_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_gt_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_hsplit_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_index_fill_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_index_select_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_le_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_linalg_matrix_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_linalg_vector_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_linspace_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_masked_fill_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_dropout_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_remainder_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_repeat_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_select_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_special_logit_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_transpose_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_addmv_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_as_strided_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_atleast_1d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_char_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_clone_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cosh_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_diagonal_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fft_irfftn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fmax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_index_add_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_lu_factor_ex_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_solve_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_tensorinv_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_median_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_minimum_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_movedim_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_native_dropout_backward_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_margin_ranking_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_max_pool2d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_reciprocal_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_repeat_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_searchsorted_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_sigmoid_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_transpose_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_triangular_solve_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_xlogy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_zeros_like_cuda, test/test_ops.py::TestCommonCUDA::test_errors_amin_cuda, test/test_ops.py::TestCommonCUDA::test_errors_copysign_cuda, test/test_ops.py::TestCommonCUDA::test_errors_diag_cuda, test/test_ops.py::TestCommonCUDA::test_errors_eq_cuda, test/test_ops.py::TestCommonCUDA::test_errors_hypot_cuda, test/test_ops.py::TestCommonCUDA::test_errors_linspace_cuda, test/test_ops.py::TestCommonCUDA::test_errors_logspace_tensor_overload_cuda, test/test_ops.py::TestCommonCUDA::test_errors_max_binary_cuda, test/test_ops.py::TestCommonCUDA::test_errors_mul_cuda, test/test_ops.py::TestCommonCUDA::test_errors_native_layer_norm_cuda, test/test_ops.py::TestCommonCUDA::test_errors_roll_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_abs_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_addcmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_asinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_ceil_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_ihfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_rfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_lu_unpack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_msort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_round_decimals_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_scatter_reduce_sum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_i0e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices___radd___cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices__unsafe_masked_index_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_all_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_angle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_as_strided_partial_views_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_atleast_2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_block_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_clamp_min_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_corrcoef_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cos_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_dsplit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_empty_permuted_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_exp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_expand_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_expand_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_expm1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_fft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_ifftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_index_fill_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_index_reduce_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_jiterator_binary_return_by_ref_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_le_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_slogdet_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_vecdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_log_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logspace_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nansum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_max_pool1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pad_replicate_negative_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_smooth_l1_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_resolve_neg_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sort_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_bessel_y1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_log_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_xlog1py_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sum_to_size_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tensordot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_trapz_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unfold_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_atan2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_count_nonzero_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_diff_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fft_ifftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_hstack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_masked_prod_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_scatter_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_modified_bessel_i0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addbmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addmm_decomposed_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_any_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_asinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atleast_1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cdouble_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cholesky_inverse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_contiguous_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_empty_permuted_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fmod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_frexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ge_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_hsplit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_int_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_jiterator_2inputs_2outputs_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_lgamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_cholesky_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_cond_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lstsq_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_matrix_rank_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_pinv_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_solve_ex_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_svdvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_tensorsolve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logical_xor_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_amin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_cumsum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_fill_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_sum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_matmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_max_binary_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_maximum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mode_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_narrow_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_narrow_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_native_batch_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_cross_entropy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_gelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_smooth_l1_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_unfold_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_norm_fro_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_normal_in_place_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_polygamma_polygamma_n_3_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_positive_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_pow_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_rad2deg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_randn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_round_decimals_0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_rsqrt_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scalar_tensor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_slice_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_hermite_polynomial_he_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_shifted_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_shifted_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_split_list_args_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_split_with_sizes_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_square_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_square_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_square_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_std_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_t_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_take_along_dim_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unbind_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unsafe_split_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_view_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_argwhere_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_diff_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_item_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_pdist_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_smooth_l1_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_permute_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_signal_windows_general_cosine_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_squeeze_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_tensor_split_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_out__refs__conversions_cfloat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs__conversions_char_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_bitwise_left_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out__refs_clamp_max_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_irfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_gt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_linalg_vector_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_linspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_new_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_real_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_reshape_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_i1e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_xlog1py_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_view_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__unsafe_masked_index_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_bitwise_and_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out_cholesky_inverse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_constant_pad_nd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cos_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_fft_hfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_fft_ifft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_hypot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_index_reduce_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_lu_factor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_tensorinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_log_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_logical_xor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_log_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_median_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nanmedian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_conv3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_ones_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_randn_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_real_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_renorm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_abs_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_addmm_decomposed_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_atanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_atanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cos_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_erf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_exp2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_expand_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_fft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_fftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_index_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_ldexp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_lu_factor_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_logspace_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_matmul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_min_binary_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nn_functional_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_scatter_reduce_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sgn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_var_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_zeros_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_resize__cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_round_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_short_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_signbit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_bessel_j1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_entr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_zeta_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_square_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_triangular_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_view_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_short_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_cumsum_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_eq_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_hfft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fmod_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_ge_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_hsplit_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_igammac_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_elu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_relu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_threshold_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_prod_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_randn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_real_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_sqrt_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_sub_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__unsafe_masked_index_put_accumulate_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_argmax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_baddbmm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_column_stack_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cross_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cumsum_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_diag_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_digamma_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_erfc_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_gather_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_imag_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_index_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_istft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_kthvalue_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_lerp_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_ldl_factor_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_mm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_new_ones_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_conv1d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_conv_transpose2d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_cosine_similarity_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_elu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_hardswish_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_local_response_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_multi_head_attention_forward_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_threshold_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_normal_number_mean_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_ones_like_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_permute_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_randint_like_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_reshape_as_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_sign_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_signbit_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_sinh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_i1_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_modified_bessel_i1_cuda, test/test_ops.py::TestCommonCUDA::test_out_xlogy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_zeros_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_pointwise_tag_coverage_cuda, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float___rdiv___cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_acos_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_acosh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_acosh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atan2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_copysign_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_exp2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_exp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_expm1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_ldexp_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_ldexp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_mvlgamma_mvlgamma_p_5_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_4_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_u_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_v_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_zeta_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sqrt_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_tan_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_tanh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_tanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_char_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_int_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_abs_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acos_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addr_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amax_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan2_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_1d_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_1d_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_3d_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_not_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_block_diag_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_tensors_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bucketize_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cat_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_chunk_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_chunk_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_chunk_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_max_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_min_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clone_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_column_stack_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_physical_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cosh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_count_nonzero_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumsum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_embed_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_embed_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_scatter_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_floor_rounding_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_no_rounding_mode_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_trunc_rounding_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dsplit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_equal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_as_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expm1_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eye_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftshift_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftshift_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flatten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flatten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fliplr_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fliplr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmax_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_frac_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_gcd_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ge_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ge_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_geometric_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_gt_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_imag_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_add_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_fill_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isclose_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isfinite_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isinf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isnan_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_le_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_le_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log10_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log10_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log1p_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_not_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_or_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_xor_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_tensor_overload_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logsumexp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_list_of_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_neg_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_strided_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nextafter_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_channel_shuffle_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_log_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_mish_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_mse_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_shuffle_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_poisson_nll_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_prelu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_prelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_relu6_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_tanhshrink_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_threshold_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_threshold_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_triplet_margin_loss_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ravel_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ravel_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ravel_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_real_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_remainder_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_repeat_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsqrt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_select_scatter_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sign_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_signbit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_bessel_j1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_erfcx_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i0e_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i0e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i1e_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_ndtr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_xlog1py_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_zeta_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_square_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sub_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tensor_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_true_divide_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vdot_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_as_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vsplit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_xlogy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_xlogy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_bitwise_right_shift_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_dstack_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_fft_hfft2_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_fft_irfft_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_neg_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_unbind_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_view_copy_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_T_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bool_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_byte_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_byte_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cdouble_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cdouble_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_chalf_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_char_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_char_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_char_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_double_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_int_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_abs_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_abs_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_add_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addr_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addr_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_alias_copy_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_amin_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_any_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_arange_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_partial_views_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_partial_views_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asinh_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atanh_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_3d_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_left_shift_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_tensors_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bucketize_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ceil_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ceil_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_max_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_max_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_column_stack_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_physical_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_contiguous_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_copysign_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cos_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cos_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cumsum_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_copy_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_digamma_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_digamma_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_digamma_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dsplit_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dsplit_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dsplit_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eq_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eq_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_equal_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_equal_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_equal_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp2_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_as_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftn_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftshift_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft2_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfftn_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfftn_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_rfft_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flatten_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flip_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flipud_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmin_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ge_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_geometric_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_i0_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_add_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_copy_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_fill_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_fill_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_select_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isclose_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isinf_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isnan_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isneginf_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_vecdot_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_vector_norm_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_and_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_and_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_xor_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_masked_fill_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_list_of_tensors_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_list_of_tensors_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_list_of_tensors_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_variadic_tensors_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_variadic_tensors_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_minimum_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mul_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_full_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_zeros_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nextafter_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hardshrink_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hardtanh_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hardtanh_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hardtanh_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_huber_loss_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pairwise_distance_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_shuffle_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_relu6_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmax_with_dtype_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmin_with_dtype_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_triplet_margin_loss_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_normal_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_pow_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_real_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reciprocal_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reciprocal_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_remainder_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_remainder_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_remainder_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_renorm_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_renorm_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_roll_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinc_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinh_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_softmax_with_dtype_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_bessel_j0_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_bessel_j0_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_erfcx_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i0e_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1e_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_softmax_with_dtype_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_softmax_with_dtype_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_5_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_5_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_spherical_bessel_j0_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_square_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sub_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_to_size_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_copy_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_to_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_to_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trace_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_copy_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_copy_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_copy_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_copy_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_copy_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_zeros_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bfloat16_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_byte_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cdouble_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cdouble_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cdouble_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cfloat_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_float_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_int_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_int_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_abs_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acos_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcdiv_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcmul_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcmul_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_alias_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_alias_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_allclose_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_amin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_arange_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_scatter_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asinh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_2d_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_2d_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_3d_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_3d_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_not_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bucketize_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clone_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_column_stack_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_column_stack_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_physical_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_physical_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_embed_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_floor_rounding_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_no_rounding_mode_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_no_rounding_mode_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_trunc_rounding_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_strided_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfinv_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expm1_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expm1_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftshift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftshift_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftshift_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ge_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_heaviside_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hstack_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_i0_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_add_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isposinf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isposinf_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_diagonal_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_svd_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_vector_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_tensor_overload_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log1p_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_softmax_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logaddexp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_tensor_overload_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_masked_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_list_of_tensors_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_list_of_tensors_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_variadic_tensors_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_minimum_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_minimum_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_movedim_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nan_to_num_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nan_to_num_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_zeros_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_celu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_l1_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pairwise_distance_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pairwise_distance_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_shuffle_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_unshuffle_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_unshuffle_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_unshuffle_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_poisson_nll_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_poisson_nll_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_poisson_nll_loss_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_relu6_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_selu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmax_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_tanhshrink_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_threshold_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_triplet_margin_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_triplet_margin_loss_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_normal_number_mean_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_positive_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_prod_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_randn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reciprocal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_remainder_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsub_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_select_scatter_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sgn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sigmoid_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sign_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_signbit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinc_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_bessel_j0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_bessel_j1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_entr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_erfcx_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_erfcx_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1e_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_logit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_1_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_5_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_xlog1py_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_xlog1py_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_zeta_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_split_with_sizes_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sqrt_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sqrt_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sub_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tanh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tensor_split_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_true_divide_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vsplit_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vstack_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bool_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bool_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_float_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_int_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_long_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_abs_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcmul_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcmul_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amax_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_any_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_arange_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_scatter_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asin_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_not_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_not_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cat_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_column_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_constant_pad_nd_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_copysign_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cos_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_count_nonzero_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumprod_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumsum_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_deg2rad_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_embed_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_floor_rounding_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_no_rounding_mode_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dsplit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eq_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_equal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_equal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfinv_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expm1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eye_cuda_float8_e4m3fnuz, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft2_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfftn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfftn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfftn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flipud_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_frac_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ge_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ge_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hypot_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_add_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_add_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_fill_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isfinite_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isnan_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isnan_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isposinf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lerp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_cross_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_cross_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_diagonal_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_diagonal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_vecdot_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_vector_norm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log1p_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_not_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_not_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_or_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_xor_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logspace_tensor_overload_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logsumexp_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logsumexp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_masked_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_variadic_tensors_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_movedim_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_neg_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_neg_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_strided_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_gelu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hardshrink_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_l1_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_layer_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_log_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_margin_ranking_loss_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_mish_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmax_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softplus_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_tanhshrink_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_threshold_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_positive_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_prod_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rad2deg_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rad2deg_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ravel_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reciprocal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_remainder_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_repeat_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_as_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_as_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rot90_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsub_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinc_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_bessel_j0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_ndtr_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_3_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_softmax_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_zeta_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_stack_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_std_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sub_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_take_along_dim_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_take_along_dim_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_take_along_dim_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tanh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tanh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tensor_split_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trunc_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unflatten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_as_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vsplit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_xlogy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_xlogy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_xlogy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___rdiv___cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___rsub___cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_atan2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_bfloat16_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_bmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cdouble_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cholesky_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_copysign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_diff_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_dist_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_erfinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_fft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_grid_sampler_2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_hypot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_item_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_cholesky_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_ldl_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_svdvals_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_tensorsolve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_log2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logical_or_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logical_xor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_fill_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_movedim_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_narrow_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_narrow_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_new_empty_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_new_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_ctc_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_rrelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_norm_nuc_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_ravel_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_reciprocal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_repeat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_resize__cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_roll_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_signal_windows_hamming_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sin_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_bessel_j1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_modified_bessel_i1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_split_list_args_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_squeeze_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sub_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_tensordot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_to_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_trace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_true_divide_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward___rmatmul___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_addcdiv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_angle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cat_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_column_stack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_conj_physical_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_dot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_ihfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_hstack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_eig_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_logsumexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_conv_transpose1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_ctc_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_linear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_prelu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_relu6_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_round_decimals_0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_round_decimals_neg_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_unflatten_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_var_mean_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_bernoulli_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_chunk_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_contiguous_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_corrcoef_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_dot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_igammac_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_isin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_eigvals_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_vander_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_logspace_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_masked_amin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_masked_argmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nanmedian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_ne_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_new_full_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_embedding_bag_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_max_unpool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_polygamma_polygamma_n_2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_reciprocal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_round_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_rsub_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_signal_windows_exponential_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_signbit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_modified_bessel_i1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_split_with_sizes_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_std_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_std_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_t_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_unbind_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_unfold_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_addbmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_addmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_bernoulli_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_cfloat_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_cumsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_double_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_empty_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_fft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_float_power_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_floor_divide_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_index_put_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_isnan_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_householder_product_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_ldl_factor_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linspace_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_normalize_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_new_empty_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_conv_transpose2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_embedding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_logsigmoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_max_unpool2d_grad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_silu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nonzero_static_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_norm_fro_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_permute_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_reciprocal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_scatter_reduce_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sqrt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_stft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_to_sparse_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_unfold_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_T_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_addmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_allclose_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_as_strided_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_char_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_cholesky_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_hfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_rfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_gt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_i0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_isneginf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_inv_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_vander_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_logaddexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_select_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_matrix_exp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_movedim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_mv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_narrow_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_ne_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_relu6_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_pow_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_roll_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_signal_windows_general_hamming_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_squeeze_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_sub_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_sum_to_size_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_byte_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_diagonal_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_dist_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_dot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_erfc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_erfinv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_exp2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_full_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_full_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_isfinite_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_eig_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_logaddexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_std_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_minimum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_msort_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nansum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_norm_inf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_bessel_y0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_entr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_modified_bessel_i1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_ndtr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_std_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_to_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_torch_ops_aten__efficient_attention_forward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_true_divide_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_unsqueeze_copy_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view_H_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_atleast_2d_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_diag_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_ifft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_float_power_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_masked_fill_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_nn_functional_l1_loss_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_nn_functional_softmin_with_dtype_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_repeat_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_unbind_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_abs_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_chunk_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_empty_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_empty_like_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_fft_ifftn_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_fft_irfft2_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_half_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_isinf_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_isreal_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_lstsq_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_lstsq_grad_oriented_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_solve_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_tensorinv_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_logical_and_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_logical_not_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_mm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_triplet_margin_loss_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_rand_like_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_renorm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_sub_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_sum_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_var_mean_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_view_as_real_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view___getitem___cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view___rsub___cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_addcdiv_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_atleast_1d_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_block_diag_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_count_nonzero_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_diagonal_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_expand_as_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fft_ifft_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_hstack_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_index_fill_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_linalg_vector_norm_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_nn_functional_pixel_shuffle_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_renorm_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_roll_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_sin_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_transpose_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_unfold_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_view_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_diagonal_scatter_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_dstack_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_expand_as_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_inner_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_isreal_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_diagonal_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_norm_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_vander_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_masked_normalize_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_new_empty_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nonzero_static_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_resize__cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_squeeze_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_unsafe_split_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view__chunk_cat_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs__conversions_bool_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs__conversions_chalf_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs__conversions_float_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_block_diag_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_diagonal_scatter_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fft_hfft_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_geometric_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_gt_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_movedim_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_softplus_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_special_log_softmax_with_dtype_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_unfold_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_addmv_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_atleast_2d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_clone_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_corrcoef_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cos_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cummin_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_diagonal_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_expand_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_geometric_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_eig_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_lu_solve_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_logical_and_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_mH_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_fill_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_matmul_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_min_reduction_with_dim_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_hardsigmoid_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_interpolate_area_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_local_response_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_mish_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_pixel_shuffle_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_softshrink_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_threshold_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_permute_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_polygamma_polygamma_n_3_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_qr_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_rand_like_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signal_windows_hann_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signal_windows_kaiser_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_split_list_args_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_squeeze_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_true_divide_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_unsqueeze_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_vstack_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake_alias_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_as_strided_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast__unsafe_masked_index_put_accumulate_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_abs_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_any_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_empty_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_empty_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_fft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_irfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_float_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_grid_sampler_2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_igammac_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_isreal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linspace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_log_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_logical_or_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_argmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_median_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_mv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_narrow_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_glu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_max_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_selu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_permute_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_reshape_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_round_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_round_decimals_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_scatter_add_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_sgn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_sinh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_zeta_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_squeeze_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_stack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_stft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_tile_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_torch_ops_aten__efficient_attention_forward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_bitwise_and_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_cfloat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_cdouble_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_div_floor_rounding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_dstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_expand_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_expand_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_ifftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_ldexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_cond_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_tensorinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_pad_reflect_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_pdist_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_rad2deg_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_reciprocal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_softmax_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_special_ndtri_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_var_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_as_strided_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_atleast_2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_complex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_erf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_expand_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_float_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_lu_unpack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_std_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_matmul_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_matrix_exp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_msort_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_mv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_polygamma_polygamma_n_0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_real_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_sinh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_t_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_unsafe_chunk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_unsafe_split_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_var_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_var_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_view_as_complex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_diag_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_diagflat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_dot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_expand_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_fft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_rfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_index_put_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_index_reduce_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_lu_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_argmin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_max_binary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nanmedian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_new_full_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_adaptive_max_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_cross_entropy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_outer_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_qr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_signal_windows_bartlett_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_signal_windows_gaussian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_sin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_hermite_polynomial_h_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_split_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_squeeze_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_t_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_trapz_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_view_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_vstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_T_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_addr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_chunk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_cummax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_cholesky_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_eig_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_householder_product_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_ldl_factor_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_tensorinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_std_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_max_binary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_cosine_similarity_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_norm_inf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_randint_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_ravel_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_sin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_torch__scaled_mm_cuda_float8_e4m3fn, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_unbind_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_unfold_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_cuda_int8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_ones_cuda_bfloat16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_ones_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_arange_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_full_cuda_int32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_zeros_cuda_int16, test/test_ops.py::TestTagsCUDA::test_tags___rmatmul___cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs__conversions_float_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_atleast_1d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_bitwise_not_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_count_nonzero_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_ihfft_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_flatten_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_geometric_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_hypot_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_i0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_imag_cuda_complex64, test/test_ops.py::TestTagsCUDA::test_tags__refs_isclose_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_lcm_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_lgamma_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_log10_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_logical_and_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_sinh_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_multigammaln_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_take_along_dim_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_trace_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_argmin_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_atanh_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_broadcast_tensors_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_clone_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_constant_pad_nd_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_cov_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_empty_permuted_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_eye_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_hfft_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_irfftn_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fmod_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_imag_cuda_complex64, test/test_ops.py::TestTagsCUDA::test_tags_istft_cuda_complex64, test/test_ops.py::TestTagsCUDA::test_tags_linalg_qr_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_log1p_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_prod_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_maximum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_mm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nansum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_native_batch_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_logsigmoid_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_randint_like_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_randn_like_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_ravel_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_resolve_conj_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_scatter_reduce_mean_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_i1e_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_zeta_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_split_list_args_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_squeeze_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_tensor_split_cuda_float32
2025-07-17T09:37:43.5306468Z 
2025-07-17T09:37:43.5306621Z Running test_ops 20/20 ... [2025-07-17 09:37:43.446741]
2025-07-17T09:37:43.5306951Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:37:43.5307841Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops.py', '-m', 'not serial', '--shard-id=20', '--num-shards=20', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:37:43.447074]
2025-07-17T09:37:59.5132455Z 
2025-07-17T09:37:59.5133258Z test_ops 4/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_4.20_2604210966282699_.log
2025-07-17T09:37:59.5640673Z Running 1698 items in this shard: test/test_ops.py::TestCommonCUDA::test_compare_cpu__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs__conversions_float_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_as_strided_partial_views_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_diagonal_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_linalg_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_select_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_copysign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_empty_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_fft_ifftshift_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_grid_sampler_2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_matrix_rank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_vector_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_soft_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_pca_lowrank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_take_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_triangular_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_unflatten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_var_mean_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_cfloat_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_diag_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_dsplit_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_index_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_index_put_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_log_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_narrow_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_narrow_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_nonzero_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_reshape_as_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_scalar_tensor_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_unfold_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes___rmatmul___cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs__conversions_char_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs__conversions_long_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_any_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_empty_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_exp2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_exp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_logaddexp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_logspace_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_logspace_tensor_overload_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_maximum_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_narrow_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_new_zeros_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_normal__in_place_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_round_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_trace_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_var_mean_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__unsafe_masked_index_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_addmm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_clamp_max_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_constant_pad_nd_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_eq_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_exp2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fft_hfft2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fft_ifft2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fft_irfft2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fft_irfft_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_ge_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_index_reduce_amax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_jiterator_4inputs_with_extra_args_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_pinv_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_pinv_hermitian_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_log1p_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_logspace_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_logspace_tensor_overload_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_mH_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_maximum_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_median_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_mode_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_mv_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_max_unpool3d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_one_hot_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_selu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_pow_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_randn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_reshape_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_round_decimals_0_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_chebyshev_polynomial_v_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_spherical_bessel_j0_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_std_unbiased_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_sum_to_size_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_t_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_trapezoid_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_view_copy_cuda, test/test_ops.py::TestCommonCUDA::test_errors_diff_cuda, test/test_ops.py::TestCommonCUDA::test_errors_fft_ifft_cuda, test/test_ops.py::TestCommonCUDA::test_errors_fmax_cuda, test/test_ops.py::TestCommonCUDA::test_errors_jiterator_binary_return_by_ref_cuda, test/test_ops.py::TestCommonCUDA::test_errors_linalg_diagonal_cuda, test/test_ops.py::TestCommonCUDA::test_errors_log_normal_cuda, test/test_ops.py::TestCommonCUDA::test_errors_lt_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_softshrink_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_triplet_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_errors_signal_windows_blackman_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_sum_layout3_cuda, test/test_ops.py::TestCommonCUDA::test_errors_special_chebyshev_polynomial_v_cuda, test/test_ops.py::TestCommonCUDA::test_errors_triu_cuda, test/test_ops.py::TestCommonCUDA::test_errors_vstack_cuda, test/test_ops.py::TestCommonCUDA::test_errors_where_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch__chunk_cat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_addmm_decomposed_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_deg2rad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_fft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_index_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_eig_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_pinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_minimum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_sign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_modified_bessel_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_angle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_argmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_argsort_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_as_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_as_strided_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_as_strided_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_atan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cdouble_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cos_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_div_floor_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_div_floor_rounding_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_empty_permuted_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_irfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_flatten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_flip_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fliplr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fmin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_int_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_jiterator_binary_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_ldexp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_le_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_svdvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_log1p_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logical_and_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logical_not_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logical_or_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logsumexp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mH_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mT_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_meshgrid_list_of_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nan_to_num_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_new_empty_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_new_empty_strided_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_celu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_channel_shuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_gaussian_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pad_constant_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_ravel_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_renorm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_scatter_reduce_amin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_scatter_reduce_sum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_signal_windows_general_hamming_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sinh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_i0e_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_modified_bessel_k0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_scaled_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_shifted_chebyshev_polynomial_u_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sub_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_torch_ops_aten__efficient_attention_forward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_trunc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unfold_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unsafe_chunk_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_var_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_zero__cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_zero__cuda_int64, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values___ror___cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_addr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_erfc_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fft_rfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_log10_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_log_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_mT_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_max_reduction_with_dim_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_nn_functional_channel_shuffle_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_reciprocal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_signbit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_split_list_args_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_split_with_sizes_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_take_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_vstack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___radd___cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___rmul___cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples__segment_reduce_offsets_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addcdiv_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_amin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_argmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atleast_1d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_broadcast_shapes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cfloat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_char_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_clamp_min_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_conj_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_count_nonzero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diagflat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_div_floor_rounding_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_empty_permuted_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_erfc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_exp2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_exp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_hfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_flip_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_gather_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_gather_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_geometric_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_isfinite_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_kthvalue_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_lerp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_ldl_factor_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_vander_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linspace_tensor_overload_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logspace_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logspace_tensor_overload_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logsumexp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_cumprod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_prod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_rms_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_triplet_margin_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_normal_in_place_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ones_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_polygamma_polygamma_n_0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_polygamma_polygamma_n_2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_rsub_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scatter_reduce_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sigmoid_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_bessel_y0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_hermite_polynomial_h_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_hermite_polynomial_h_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_i0e_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_split_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_split_list_args_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_squeeze_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_squeeze_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tensor_split_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_to_sparse_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_transpose_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tril_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_triu_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unbind_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_addbmm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_argwhere_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_clone_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_item_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_pairwise_distance_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_repeat_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_signal_windows_bartlett_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_where_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_out__batch_norm_with_update_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs__conversions_half_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs__conversions_long_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs__conversions_polar_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_bitwise_not_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out__refs_bitwise_or_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out__refs_clamp_min_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_conj_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_div_floor_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_exp2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_isfinite_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_logspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_movedim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_ne_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_new_empty_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_reciprocal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_rsqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_sqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_std_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_view_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_argsort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_as_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cummax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_fft_fftshift_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_frexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_full_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_igamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_index_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_isinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_inv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_inv_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_ldl_factor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_lstsq_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_log_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_lu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_softmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_new_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_mish_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_acosh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_addmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cumprod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_ifft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_float_power_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fmod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_frac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_hstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_det_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_eigvalsh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_log10_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_max_binary_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nn_functional_gelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sigmoid_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_squeeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_reshape_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_rot90_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_scalar_tensor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_scatter_reduce_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_modified_bessel_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_std_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_take_along_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_vdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_complex_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_amin_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_block_diag_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_chunk_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_cosh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_erfc_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_exp_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_ihfft2_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_irfftn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_index_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_log1p_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_logaddexp_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_channel_shuffle_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_layer_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_tanhshrink_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_permute_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_pow_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_ravel_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_sinh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_ndtr_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_std_mean_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_unflatten_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__upsample_bilinear2d_aa_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_addcdiv_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_angle_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_any_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_bincount_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_diff_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_ifft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_ihfftn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_rfftn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_float_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_geometric_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_log_softmax_with_dtype_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_mH_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_cumsum_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_normalize_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_mul_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nanmean_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_native_dropout_backward_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_new_zeros_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_adaptive_avg_pool3d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_dropout_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_gaussian_nll_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_instance_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_l1_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_layer_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_max_pool1d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_multi_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_multilabel_soft_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_softsign_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_qr_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_sigmoid_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_sinc_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_airy_ai_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_chebyshev_polynomial_t_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_to_sparse_cuda, test/test_ops.py::TestCommonCUDA::test_out_zero__cuda_float32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atanh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_cos_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_digamma_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erf_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erfinv_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_exp2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log10_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log10_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_logit_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_logit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_masked_var_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_0_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_3_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_rad2deg_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_hermite_polynomial_h_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_t_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_v_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_tanh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_true_divide_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_true_divide_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cfloat_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cfloat_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_double_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_double_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_short_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_abs_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_abs_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acos_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acos_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acosh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_alias_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_all_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_all_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_any_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_arange_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asinh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asinh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_1d_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_2d_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_3d_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_not_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_right_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_tensors_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cauchy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ceil_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_max_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clone_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_physical_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_constant_pad_nd_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_constant_pad_nd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_constant_pad_nd_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_contiguous_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_contiguous_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_contiguous_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_copysign_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_copysign_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_deg2rad_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_embed_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_like_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eq_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erf_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expm1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eye_cuda_float8_e4m3fnuz, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eye_cuda_float8_e5m2fnuz, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftshift_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfftn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft2_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfftn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flatten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flip_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flip_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_divide_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_divide_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmin_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_geometric_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_geometric_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_gt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_gt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_gt_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hsplit_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hypot_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_i0_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isclose_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isclose_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isinf_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isposinf_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_item_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_le_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_vecdot_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_tensor_overload_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log10_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_and_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_and_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_not_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_xor_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_tensor_overload_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_tensor_overload_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_tensor_overload_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_list_of_tensors_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_minimum_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_movedim_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_movedim_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_native_layer_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ne_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ne_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_strided_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_hardtanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_huber_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_leaky_relu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_log_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_margin_ranking_loss_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_mse_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_nll_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pairwise_distance_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_shuffle_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_shuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_unshuffle_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softplus_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_threshold_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_positive_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_positive_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rad2deg_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ravel_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reciprocal_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_repeat_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rot90_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsqrt_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sigmoid_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sigmoid_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinc_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinc_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_softmax_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i0e_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i0e_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_5_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_ndtr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_xlog1py_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_zeta_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_square_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tensor_split_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tensor_split_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tensor_split_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trace_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_indices_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_triu_indices_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_var_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_as_complex_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vsplit_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vsplit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_xlogy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs__conversions_complex_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_amin_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_heaviside_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_maximum_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_rot90_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_rsub_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_tril_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_view_as_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_T_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bool_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_chalf_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_float_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_half_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_int_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_int_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_long_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_abs_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_add_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcmul_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_all_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_amax_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_arange_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_arange_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_copy_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_partial_views_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_scatter_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_scatter_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asinh_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan2_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atanh_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_or_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_right_shift_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_xor_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_xor_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_block_diag_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_tensors_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_chunk_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_max_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_max_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_min_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_copysign_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cos_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cosh_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cosh_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cosh_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_count_nonzero_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_count_nonzero_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_count_nonzero_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_embed_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_digamma_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_digamma_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dot_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dsplit_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dstack_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_like_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eq_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erf_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp2_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_as_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expm1_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft2_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfftn_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftshift_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfftn_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_rfft_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_rfftn_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flip_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_float_power_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_divide_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmax_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmin_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmin_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmod_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmod_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_geometric_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_gt_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_heaviside_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hstack_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_i0_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_copy_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_copy_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_fill_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_select_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isinf_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isnan_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isreal_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isreal_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_item_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lgamma_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_cross_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_diagonal_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_norm_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_norm_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_tensor_overload_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log2_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_normal_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logaddexp2_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logaddexp2_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logaddexp_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_and_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_or_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logspace_tensor_overload_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logsumexp_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_list_of_tensors_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_list_of_tensors_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_minimum_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_movedim_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_movedim_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mul_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nan_to_num_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_copy_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_full_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_full_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nextafter_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_l1_loss_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_leaky_relu_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_margin_ranking_loss_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_nll_loss_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_unshuffle_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_selu_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_normal_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rad2deg_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_randn_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_real_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_remainder_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_renorm_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_repeat_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_as_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_as_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_roll_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsqrt_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sigmoid_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sign_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinh_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_erfcx_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i0e_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1e_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1e_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1e_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_ndtr_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_1_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_1_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtr_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_zeta_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_square_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_copy_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_stack_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_std_mean_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_stft_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_to_size_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_take_along_dim_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tan_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tensor_split_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_to_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_to_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_copy_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_copy_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vdot_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vstack_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_xlogy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_byte_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cdouble_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cfloat_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_double_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_long_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_abs_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acos_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acos_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcmul_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_alias_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_all_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_amax_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_amin_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_any_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_any_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_arange_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_block_diag_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cat_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cat_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ceil_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_max_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_min_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_min_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_column_stack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_copysign_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cos_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cosh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cosh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_count_nonzero_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_count_nonzero_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_scatter_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dsplit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_strided_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfc_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfinv_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_as_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fft_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fft_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flip_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flip_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_float_power_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_float_power_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_divide_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_divide_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_add_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isclose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isclose_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isnan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_istft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_le_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_le_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_cross_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log10_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log1p_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_not_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_or_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_or_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logsumexp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logsumexp_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logsumexp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_masked_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_maximum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_maximum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mean_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mean_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_minimum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ne_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ne_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ne_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_neg_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_neg_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_neg_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_full_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_ones_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_channel_shuffle_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_gelu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_margin_ranking_loss_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_mish_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pairwise_distance_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_poisson_nll_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_poisson_nll_loss_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_prelu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_smooth_l1_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmin_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmin_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ones_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ones_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_repeat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_select_scatter_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sgn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sgn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sigmoid_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sigmoid_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sign_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sign_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sign_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_signbit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_entr_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_erfcx_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i0e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i0e_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1e_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1e_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_softmax_with_dtype_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_logit_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_3_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_3_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_5_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_spherical_bessel_j0_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_split_with_sizes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_std_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_to_size_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_take_along_dim_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_take_along_dim_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tan_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tensor_split_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_to_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_to_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_indices_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_true_divide_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_true_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_true_divide_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_as_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_as_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_zeros_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bool_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bool_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_double_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_float_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_half_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_abs_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acosh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_add_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcdiv_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_any_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_any_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asin_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_or_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bucketize_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cat_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_max_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_min_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_constant_pad_nd_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_constant_pad_nd_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_count_nonzero_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumprod_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumprod_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_deg2rad_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_embed_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_digamma_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_no_rounding_mode_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_like_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfc_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfinv_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp2_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_as_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eye_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft2_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifftn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfftn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flip_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_float_power_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_divide_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_select_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isfinite_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isfinite_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isnan_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isneginf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isposinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isreal_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lcm_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_le_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lerp_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_cross_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_cross_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_matrix_norm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_vecdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log10_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_not_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_xor_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_xor_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_xor_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_maximum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_variadic_tensors_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_minimum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_mul_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ne_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_zeros_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_channel_shuffle_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_channel_shuffle_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_l1_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_layer_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_log_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pairwise_distance_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_shuffle_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_shuffle_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_shuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_unshuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_poisson_nll_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmin_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_tanhshrink_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_triplet_margin_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ravel_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_remainder_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsqrt_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsqrt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsub_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sgn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sgn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinc_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i0e_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_logit_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_logit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_5_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_5_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_xlog1py_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_xlog1py_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sqrt_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_stack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_std_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_stft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_to_size_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_take_along_dim_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_take_along_dim_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tan_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tan_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tensor_split_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trace_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_as_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vsplit_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vsplit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vstack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_xlogy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_zeros_cuda_int8, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___rdiv___cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_as_strided_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cdist_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cholesky_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cov_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cumprod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_diagonal_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_dstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_empty_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_eq_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_irfft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_half_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_isfinite_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_isinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_jiterator_unary_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_inv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_lstsq_grad_oriented_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_lu_factor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_solve_triangular_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_tensorinv_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linspace_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logspace_tensor_overload_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_lu_unpack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_std_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_narrow_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_new_zeros_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_celu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_silu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_polygamma_polygamma_n_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_quantile_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_reciprocal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_reshape_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_rsub_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_select_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_signal_windows_hann_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_shifted_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_svd_lowrank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_tan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_trace_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_trapezoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unflatten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_var_mean_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_zero__cuda_complex64, test/test_ops.py::TestCompositeComplianceCUDA::test_backward___rmod___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_addcmul_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_chalf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cumsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_digamma_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_dstack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_flatten_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_flip_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_frac_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_index_reduce_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_matrix_power_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_tensorsolve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_cumprod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_grid_sample_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_remainder_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_repeat_interleave_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_round_decimals_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_sqrt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input___rmatmul___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input__segment_reduce_offsets_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_addmm_decomposed_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_atleast_2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_bmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_cauchy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_clamp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_conj_physical_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_erf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_erfc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fliplr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_float_power_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_isnan_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_eig_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_lu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_matrix_rank_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_svdvals_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_log_normal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_log_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_masked_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nextafter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_embedding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nonzero_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_normal_number_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_permute_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_reshape_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_rot90_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_signal_windows_kaiser_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_entr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_hermite_polynomial_h_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_i1e_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_xlog1py_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_sqrt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_sub_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_var_mean_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_xlogy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_zeros_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_cosh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_dot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_eq_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_erf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_fftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_irfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fill_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fliplr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_geometric_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_hstack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_qr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_tensorinv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_vector_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_logspace_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_new_zeros_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_conv_transpose1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_max_unpool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_soft_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_normal_number_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_entr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_modified_bessel_i0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_zeta_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_squeeze_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_squeeze_multiple_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_unique_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_vsplit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_vstack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_xlogy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator__chunk_cat_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_all_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_amin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_aminmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_any_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_dsplit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_hfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_ihfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_ihfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_gradient_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_lgamma_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linspace_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_log10_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_log_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_amax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_new_empty_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_interpolate_nearest-exact_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_pinverse_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_polygamma_polygamma_n_2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_qr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_rad2deg_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_bessel_j1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_torch_ops_aten__efficient_attention_forward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay__batch_norm_with_update_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay__unsafe_masked_index_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_addmv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_as_strided_partial_views_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_asin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_atleast_2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_expand_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_ifftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_rfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_geqrf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_index_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_index_fill_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_lu_factor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_matrix_power_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_log1p_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_log2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_fill_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_log_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_embedding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_interpolate_nearest-exact_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_interpolate_nearest_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_max_pool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_relu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_renorm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_reshape_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_hermite_polynomial_h_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_ndtri_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_spherical_bessel_j0_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_cfloat_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_int_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_expm1_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_irfft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fill_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_flatten_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_linspace_tensor_overload_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_meshgrid_list_of_tensors_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_ne_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_new_ones_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_norm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_stack_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_std_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_std_mean_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_tensor_split_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__unsafe_masked_index_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_as_strided_scatter_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_combinations_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_contiguous_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_empty_permuted_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_expand_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_lu_factor_ex_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_slogdet_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_log10_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_lu_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_conv3d_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_normalize_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_softsign_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_pca_lowrank_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_short_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_split_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_tensordot_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_unsafe_split_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view___rdiv___cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fft_fftshift_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fft_hfft_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_index_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_linalg_svd_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_square_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_sum_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_addmv_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_atan_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_block_diag_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_diagonal_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_double_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_exp_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fft_fft_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_index_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_lerp_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_ldl_factor_ex_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_mean_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nansum_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_neg_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_new_full_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_conv_transpose3d_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_l1_loss_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_take_along_dim_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_transpose_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_vsplit_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_atan_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_conj_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_conj_physical_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fmod_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_frac_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_ge_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_hsplit_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_logical_not_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_new_zeros_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_softmax_with_dtype_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_threshold_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_rsqrt_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_sinh_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_view_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cfloat_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_chalf_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cholesky_inverse_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_div_trunc_rounding_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fill_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_qr_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_vector_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_logsumexp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_median_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_sum_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_mv_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_narrow_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_conv2d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_dropout3d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_gaussian_nll_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_hinge_embedding_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_interpolate_linear_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_max_pool1d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_selu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_randn_like_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_scatter_reduce_amin_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_legendre_polynomial_p_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_to_sparse_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_triangular_solve_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_var_unbiased_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake_allclose_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_any_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast___radd___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast___rmul___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_acosh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_atanh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cholesky_inverse_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cumsum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_deg2rad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_div_floor_rounding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_rfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_flip_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_flipud_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_floor_divide_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_gather_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_hsplit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_index_reduce_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_isclose_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_jiterator_binary_return_by_ref_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_lstsq_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_lu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_matrix_rank_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_svd_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_tensorinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_vector_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_mode_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_movedim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_multinomial_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_outer_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_positive_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_randint_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_round_decimals_0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_round_decimals_neg_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_sigmoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_i1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_log_ndtr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_square_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_std_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_svd_lowrank_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_take_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_triangular_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_view_as_complex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_bitwise_xor_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_bool_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_cos_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_cross_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp___radd___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp___rmatmul___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp___rmul___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp__unsafe_masked_index_put_accumulate_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_atleast_3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_bmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_corrcoef_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_cross_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_irfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_i0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_kron_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_pinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_vecdot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_logit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_masked_softmin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nanmean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_elu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_normalize_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_smooth_l1_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_norm_inf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_normal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_pow_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_ravel_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_round_decimals_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_sgn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_slice_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_slice_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_special_log_ndtr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp___rmatmul___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_addcmul_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_broadcast_to_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_clone_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_diff_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_einsum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_rfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fliplr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_index_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_kron_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_cross_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_native_layer_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_gelu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_interpolate_nearest-exact_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_max_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_max_unpool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_normal_number_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_round_decimals_neg_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_sort_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_view_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_xlogy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_empty_strided_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_expm1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_igamma_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_jiterator_2inputs_2outputs_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_det_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_lu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_matrix_power_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_vander_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logical_or_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_sum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_var_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_embedding_bag_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_gelu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_kl_div_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_ones_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_ormqr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_scatter_reduce_sum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_signal_windows_blackman_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_sub_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_t_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_topk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_torch__scaled_mm_cuda_float8_e4m3fn, test/test_ops.py::TestFakeTensorCUDA::test_fake_torch_ops_aten__efficient_attention_forward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_unbind_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_as_strided_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_baddbmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_chalf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_exp2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_irfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fliplr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_index_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_index_reduce_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_istft_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_logit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_logsumexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_argmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_normalize_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_native_layer_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_adaptive_avg_pool1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_kl_div_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_pairwise_distance_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_rand_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_round_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_signal_windows_kaiser_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_signbit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_i1e_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_split_with_sizes_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_tril_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_triu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_arange_cuda_float16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_tensor_overload_cuda_int32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_tensor_overload_cuda_int32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_ones_cuda_complex128, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags___getitem___cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_cauchy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_conj_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_expand_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_fftshift_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_rfft2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fmin_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_le_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_log1p_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_spherical_bessel_j0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_squeeze_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_tril_indices_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_true_divide_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__unsafe_masked_index_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_angle_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_bitwise_or_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags_cdouble_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_complex_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_ifftshift_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_flipud_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_histc_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_hypot_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_lerp_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_det_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_lu_solve_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_slogdet_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_lt_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_lu_solve_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_sum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_msort_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_glu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_renorm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_signal_windows_general_hamming_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_split_with_sizes_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_tile_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_torch_ops_aten__flash_attention_forward_cuda_float16, test/test_ops.py::TestTagsCUDA::test_tags_transpose_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_transpose_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_tril_indices_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags_triu_indices_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags_unflatten_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_unfold_copy_cuda_float32
2025-07-17T09:37:59.6080700Z 
2025-07-17T09:37:59.6080879Z Running test_ops_gradients 3/3 ... [2025-07-17 09:37:59.528185]
2025-07-17T09:37:59.6081441Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:37:59.6082351Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_ops_gradients.py', '-m', 'not serial', '--shard-id=3', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:37:59.528585]
2025-07-17T09:38:00.6151887Z 
2025-07-17T09:38:00.6153079Z test_ops 7/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_7.20_da8c2cec72a8d8d5_.log
2025-07-17T09:38:00.6644267Z Running 1694 items in this shard: test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_addcmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_expand_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_logspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_mul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_new_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_rsub_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_vdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_div_floor_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_gradient_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_eigvalsh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_svdvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_lu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_masked_cumprod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_mul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_max_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_rrelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_resolve_conj_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_slice_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_special_zeta_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_std_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_zeros_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_exp_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_nn_functional_conv1d_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_sgn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_sin_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_unbind_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_T_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_all_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_bitwise_and_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_bitwise_not_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_ihfft_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_irfft_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_rfft2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_linalg_svd_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_channel_shuffle_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_leaky_relu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_pairwise_distance_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_smooth_l1_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_signbit_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_special_i0e_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_split_with_sizes_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_sqrt_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_squeeze_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_std_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_aminmax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_any_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_bernoulli_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_bitwise_left_shift_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_bmm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_broadcast_to_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_dsplit_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_grid_sampler_2d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_hstack_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_hypot_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_i0_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_ldexp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_logdet_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_std_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_var_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_embedding_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_hinge_embedding_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_interpolate_area_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_max_unpool2d_grad_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_pad_circular_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_polygamma_polygamma_n_4_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_randint_like_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_resize__cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_sinc_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_sinh_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_i1_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_xlog1py_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_svd_lowrank_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_torch_ops_aten__efficient_attention_forward_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_torch_ops_aten__flash_attention_forward_cuda, test/test_ops.py::TestCommonCUDA::test_errors_hsplit_cuda, test/test_ops.py::TestCommonCUDA::test_errors_logcumsumexp_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_avg_pool2d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_hardtanh_cuda, test/test_ops.py::TestCommonCUDA::test_errors_ormqr_cuda, test/test_ops.py::TestCommonCUDA::test_errors_reshape_as_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_mul_layout0_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_zeros_like_layout2_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_all_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_clamp_min_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_diagonal_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_digamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_irfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_nansum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_nonzero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_norm_fro_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_scatter_reduce_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_laguerre_polynomial_l_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_modified_bessel_k0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_square_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_take_along_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_transpose_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices___radd___cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_addr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_aminmax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cauchy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cfloat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_column_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cummax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_double_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_erf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_erf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_eye_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_ifftshift_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_ihfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_rfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_grid_sampler_2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_heaviside_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_item_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_jiterator_4inputs_with_extra_args_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_cholesky_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_vander_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logical_or_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_amin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_cumsum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_new_ones_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_selu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_unfold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nonzero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_ormqr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_pinverse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_resize__cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_resize_as__cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_rsub_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_scalar_tensor_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_ndtri_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_stack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_t_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tan_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_trace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_triangular_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unflatten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unsafe_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unsqueeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_view_as_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_acos_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_add_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_contiguous_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_equal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fmin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_full_like_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_gt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_jiterator_unary_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_meshgrid_list_of_tensors_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_nan_to_num_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_reshape_as_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_rsqrt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_scaled_modified_bessel_k0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_split_with_sizes_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_view_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_H_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_T_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___ror___cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_asin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bfloat16_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bincount_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cartesian_prod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_chalf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cumprod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_empty_strided_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_expand_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_fft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_fft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_rfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_gradient_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_reduce_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_isinf_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_isneginf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_jiterator_unary_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_cholesky_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_det_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_diagonal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_eigvalsh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_matrix_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_svdvals_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logical_and_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_argmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_select_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_var_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_var_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_max_reduction_with_dim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_median_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_meshgrid_list_of_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mv_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nanmean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_native_dropout_backward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_new_zeros_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_conv_transpose1d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_cosine_similarity_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_feature_alpha_dropout_without_train_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_hardtanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_unfold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_norm_inf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_outer_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_permute_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_polygamma_polygamma_n_4_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_rand_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_randint_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_reshape_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_round_decimals_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_rsub_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scalar_tensor_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_signal_windows_kaiser_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_i1e_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_shifted_chebyshev_polynomial_u_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_split_with_sizes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_transpose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_trapezoid_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_var_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_var_unbiased_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_vdot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_view_as_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_zero__cuda_float32, test/test_ops.py::TestCommonCUDA::test_numpy_ref_diagflat_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_diagflat_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_permute_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_squeeze_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_view_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out___rand___cuda_int64, test/test_ops.py::TestCommonCUDA::test_out__refs__conversions_cdouble_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs__conversions_float_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_block_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_diagonal_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_fft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_rfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_flipud_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_permute_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_remainder_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_rot90_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_argwhere_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_bmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_chalf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_dist_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_integral_dtype__refs_prod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_out_jiterator_4inputs_with_extra_args_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_multi_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_max_binary_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nansum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_grid_sample_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_max_pool1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_max_unpool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_quantile_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_acos_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_add_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_addcmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cholesky_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_diagonal_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_hfft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_irfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_lu_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_logcumsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_msort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_mv_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_outer_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_rsqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_tensordot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_true_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_vdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_scaled_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_trapezoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_float_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_long_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_polar_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_addr_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_allclose_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_any_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_as_strided_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_atleast_1d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_constant_pad_nd_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_fftshift_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_minimum_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_positive_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_rad2deg_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_logit_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_stack_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_t_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_tanh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_zeros_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_as_strided_partial_views_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_bfloat16_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_broadcast_to_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_clamp_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cumulative_trapezoid_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_einsum_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_empty_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_irfft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_ge_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_index_reduce_amax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_kron_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_pinv_singular_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_ne_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_new_full_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_adaptive_avg_pool1d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_interpolate_nearest_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_max_unpool1d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_softplus_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_softshrink_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_ones_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_polygamma_polygamma_n_3_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_prod_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_randint_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_resolve_conj_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_signal_windows_general_cosine_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_legendre_polynomial_p_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_modified_bessel_k1_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_spherical_bessel_j0_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_stack_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_std_unbiased_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_stft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_torch_ops_aten__safe_softmax_default_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_triu_indices_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_trunc_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_unsafe_chunk_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_unsafe_split_cuda, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_asin_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_asinh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atan2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atan_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_cosh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_exp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_expm1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_i0_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log1p_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_logit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_1_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sigmoid_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sinc_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_hermite_polynomial_he_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_u_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_v_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_tan_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_tan_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_true_divide_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_byte_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cdouble_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_char_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_complex_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_double_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_float_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_half_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_half_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_long_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcdiv_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_alias_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_alias_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_any_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_arange_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_partial_views_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_scatter_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asinh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asinh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asinh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_3d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_or_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_block_diag_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_block_diag_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_tensors_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ceil_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_chunk_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_chunk_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_max_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_min_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clone_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_column_stack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_physical_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_physical_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_copysign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_count_nonzero_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_count_nonzero_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumprod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumsum_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_embed_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dot_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eq_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erfinv_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp2_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exponential_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eye_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftshift_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftshift_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftshift_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft2_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfftn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flatten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fliplr_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fliplr_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_float_power_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_divide_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmax_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmin_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_frac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_frexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_frexp_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hsplit_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hypot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isclose_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isclose_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_le_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_le_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lerp_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_norm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_svd_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_vecdot_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_vector_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_tensor_overload_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_xor_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_xor_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_maximum_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_minimum_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_movedim_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_movedim_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_movedim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_strided_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_alpha_dropout_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_alpha_dropout_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_log_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_nll_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_selu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmax_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmin_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmin_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_normal__in_place_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ones_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rad2deg_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ravel_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_real_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_real_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_remainder_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_repeat_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_as_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_as_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_roll_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rot90_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_select_scatter_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_select_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sign_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_signbit_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_signbit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinc_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_erfcx_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_1_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_3_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_3_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_ndtr_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_std_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sub_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_to_size_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tan_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tensor_split_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_triu_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_triu_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_var_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_var_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_as_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vsplit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vsplit_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vsplit_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_eq_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_fft_fft2_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_fft_irfftn_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_nextafter_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_vsplit_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_T_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bool_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_char_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_complex_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_double_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_double_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_float_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_float_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_long_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_add_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcmul_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcmul_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addr_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_alias_copy_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_all_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_all_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_allclose_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_partial_views_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_scatter_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_block_diag_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cat_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_chunk_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_min_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_copysign_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_deg2rad_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_embed_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_scatter_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfc_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfc_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfinv_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfinv_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfinv_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfinv_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_as_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_as_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eye_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eye_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft2_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftn_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftshift_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft2_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfftn_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_rfft_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fill_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flip_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmod_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_frac_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ge_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_geometric_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hypot_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_add_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_fill_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_fill_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isfinite_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isfinite_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isposinf_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_diagonal_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_matrix_norm_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_svd_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_svdvals_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_tensor_overload_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log1p_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log1p_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log2_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_masked_fill_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_masked_fill_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_maximum_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_movedim_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mul_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mul_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nan_to_num_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_copy_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ne_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_full_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_ones_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_ones_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_zeros_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_zeros_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_dropout_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_glu_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_glu_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_glu_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hinge_embedding_loss_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_leaky_relu_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_log_softmax_with_dtype_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_mse_loss_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pairwise_distance_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_shuffle_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_prelu_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmin_with_dtype_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_tanhshrink_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_tanhshrink_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_normal_number_mean_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_real_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_real_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reciprocal_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reciprocal_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_remainder_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_repeat_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_as_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_as_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_roll_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_round_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_round_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsub_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsub_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sigmoid_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_signbit_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sin_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_softmax_with_dtype_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1e_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_ndtr_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_softmax_with_dtype_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_logit_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtr_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_xlog1py_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_multiple_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_multiple_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_multiple_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_stack_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sub_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sub_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sub_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_copy_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tan_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tanh_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tensor_split_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_to_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trace_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tril_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trunc_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unflatten_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vsplit_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_zeros_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_byte_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cdouble_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cdouble_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_float_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_half_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_int_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_short_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_abs_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_abs_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcmul_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_all_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_allclose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_allclose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_amin_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_and_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_and_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_to_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_to_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bucketize_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bucketize_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cat_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cauchy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_min_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clone_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_physical_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_physical_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_copysign_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cos_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_deg2rad_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_digamma_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_digamma_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_trunc_rounding_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_like_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_strided_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eq_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_equal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erf_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_as_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expm1_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_float8_e4m3fn, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_float8_e5m2, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft2_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifft2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifft_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft2_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfftn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flatten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flatten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_divide_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmod_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gcd_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_heaviside_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hstack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_add_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_select_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isclose_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isinf_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isnan_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isnan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_istft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_le_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_le_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_svdvals_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_vector_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_tensor_overload_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log10_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log1p_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_softmax_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_xor_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_tensor_overload_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_tensor_overload_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logsumexp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_masked_fill_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_masked_fill_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_masked_fill_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_maximum_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_variadic_tensors_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_variadic_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_movedim_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_movedim_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mul_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ne_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_neg_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_full_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_full_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_ones_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_zeros_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_zeros_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_alpha_dropout_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_gelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hardtanh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_l1_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_l1_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_leaky_relu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_log_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_log_softmax_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_nll_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_unshuffle_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_prelu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_selu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softshrink_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_positive_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rad2deg_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rad2deg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_real_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_remainder_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_roll_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_round_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsub_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsub_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_select_scatter_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_select_scatter_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sgn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sgn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sigmoid_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_bessel_j1_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_entr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1e_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_softmax_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_logit_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_std_mean_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_take_along_dim_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tan_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tanh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trace_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trace_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_triu_indices_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trunc_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unflatten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vsplit_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_zeros_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bfloat16_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cdouble_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cfloat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_chalf_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_char_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_complex_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_double_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_float_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_float_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_short_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_add_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addr_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_all_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amax_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_partial_views_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_scatter_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asin_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_1d_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_and_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_block_diag_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_to_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cat_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cat_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_chunk_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_column_stack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_physical_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_physical_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_physical_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_constant_pad_nd_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cos_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_count_nonzero_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumprod_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumsum_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_embed_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_floor_rounding_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dsplit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dstack_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dstack_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_like_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_like_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eq_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eq_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erf_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfc_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfc_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfc_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eye_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfftn_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flatten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flipud_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_float_power_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_float_power_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmax_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ge_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ge_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_geometric_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hsplit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hsplit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hypot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_select_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isneginf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isreal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_diagonal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_vector_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log1p_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logaddexp2_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_and_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_not_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logsumexp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lt_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_maximum_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_variadic_tensors_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_minimum_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_native_layer_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_native_layer_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ne_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_full_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_zeros_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_group_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hardshrink_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hardtanh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_margin_ranking_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_tanhshrink_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_triplet_margin_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_normal__in_place_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_normal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_prod_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ravel_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reciprocal_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_repeat_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_as_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_as_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_select_scatter_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_select_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_bessel_j0_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1e_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_ndtr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_logit_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_logit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_1_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_3_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_5_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_5_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_spherical_bessel_j0_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_split_with_sizes_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sqrt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_square_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_multiple_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_to_size_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_to_size_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tril_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_true_divide_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trunc_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unflatten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unflatten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_var_mean_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_xlogy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_addmm_decomposed_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_allclose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_diagflat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_eq_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_fftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_hfftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_irfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_geqrf_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_igamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_imag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_isfinite_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_isnan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_isnan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_inv_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_qr_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_solve_ex_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_matrix_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_mul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_multinomial_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nanmean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_narrow_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_native_layer_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pad_reflect_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_remainder_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_scatter_reduce_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_signal_windows_blackman_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_signal_windows_exponential_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_signal_windows_gaussian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_std_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sub_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_svd_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_tan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_tanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_tensor_split_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_topk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_trunc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unsqueeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_var_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_zeros_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_H_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_T_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward__unsafe_masked_index_put_accumulate_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_broadcast_to_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_contiguous_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_diagonal_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_index_select_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_cond_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_svd_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_logaddexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_mH_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_softmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_cosine_similarity_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_interpolate_bilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_max_unpool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_repeat_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_sinc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_special_xlog1py_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_tensor_split_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input___rmul___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_addmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_as_strided_partial_views_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_byte_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_complex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_deg2rad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_hypot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_isfinite_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_pinv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_masked_logaddexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_masked_median_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_native_dropout_backward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_normalize_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_outer_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_randint_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_roll_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_round_decimals_neg_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_scatter_reduce_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_unflatten_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_zero__cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad___rdiv___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad___rmul___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad__unsafe_masked_index_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_allclose_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_amin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_as_strided_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_as_strided_partial_views_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_atan2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_cross_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_digamma_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_lgamma_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_ldl_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_log_normal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_logcumsumexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_cumprod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_mul_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_mv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nanmedian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_native_dropout_backward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_pad_reflect_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_normal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_randn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_reshape_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sinc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_spherical_bessel_j0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_split_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_triangular_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator___rmul___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_argmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_bfloat16_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_bmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_conj_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_copysign_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_diagflat_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_irfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_histc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_matrix_rank_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_svdvals_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_logsumexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_long_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_lu_unpack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_softmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_std_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_matmul_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_glu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_instance_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_kl_div_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_pad_reflect_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_prelu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_selu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_silu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_polar_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_resize__cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_round_decimals_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_rsqrt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_i1e_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_stack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_stft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_tensor_split_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_unique_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay___getitem___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_argsort_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_combinations_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_deg2rad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_fft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_float_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_i0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_pinv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_qr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_tensorsolve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_log_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_logical_xor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_lu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_cumprod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_select_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_conv3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_max_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_normalize_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_round_decimals_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_bessel_j1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_split_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_stft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_take_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_triangular_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_trunc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_unfold_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_unsqueeze_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view___rsub___cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_chunk_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_constant_pad_nd_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_cos_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_imag_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_neg_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_nn_functional_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_nn_functional_pairwise_distance_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_positive_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_sgn_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_sinh_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_t_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_vsplit_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_acos_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_addcdiv_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_byte_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_ldl_factor_ex_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_lu_solve_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_vander_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_vecdot_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_log2_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_logdet_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nanmean_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_roll_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_sum_to_size_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_trapz_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_tril_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_triu_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_unsqueeze_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_T_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs__conversions_bfloat16_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs__conversions_byte_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs__conversions_cdouble_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_add_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_as_strided_scatter_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_column_stack_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_conj_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_diag_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_diagonal_scatter_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_exp_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_nn_functional_pairwise_distance_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_sum_to_size_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_t_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_unsqueeze_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_var_mean_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_atleast_1d_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_dist_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_dsplit_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_expand_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_flip_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_float_power_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_index_put_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_isinf_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_item_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_cholesky_ex_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_pinv_singular_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_slogdet_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_logspace_tensor_overload_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_mT_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_prod_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_svd_lowrank_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_t_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_tan_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_transpose_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_unflatten_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_var_mean_unbiased_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_add_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_amin_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_eq_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_log10_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_logaddexp2_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_group_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_huber_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_ravel_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_reciprocal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_take_along_dim_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_to_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_allclose_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_flatten_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_flip_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_full_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_ge_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_hstack_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_index_reduce_mean_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_lerp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_log1p_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_logcumsumexp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_logical_not_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_softmin_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nanmean_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_ne_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_adaptive_avg_pool1d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_binary_cross_entropy_with_logits_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_embedding_bag_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_feature_alpha_dropout_without_train_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_pad_reflect_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_soft_margin_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_tanhshrink_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_upsample_bilinear_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_permute_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_pow_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_round_decimals_0_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signal_windows_general_cosine_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_chebyshev_polynomial_v_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_ndtri_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_tril_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_unsqueeze_copy_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake___radd___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake__chunk_cat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_angle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_asinh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast___getitem___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast__chunk_cat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_arange_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_bitwise_xor_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_ceil_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_clamp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_combinations_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cumprod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_diff_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_erfc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_expand_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_rfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_isneginf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_jiterator_2inputs_2outputs_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_jiterator_unary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_cholesky_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_solve_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_mT_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_mm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_neg_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_new_empty_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_interpolate_bilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_max_unpool3d_grad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_normalize_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nonzero_static_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_randint_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_reshape_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signal_windows_cosine_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_slice_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_bessel_y1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_squeeze_multiple_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_torch_ops_aten__flash_attention_forward_cuda_float16, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_transpose_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_zeros_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_clamp_min_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp___getitem___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp__segment_reduce_offsets_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_baddbmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_broadcast_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_conj_physical_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_constant_pad_nd_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_cummin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_diag_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_diagonal_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_hfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_hsplit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_hstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_lstsq_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_lu_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_mul_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_softplus_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_roll_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_rot90_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_rsqrt_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_sum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_topk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_tril_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_true_divide_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp__segment_reduce_offsets_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp__unsafe_masked_index_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_acosh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_alias_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_asinh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_column_stack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cos_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cross_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cumprod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_diagflat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_diagonal_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_diagonal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_ifftshift_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_flipud_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_det_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_householder_product_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_lu_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nan_to_num_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_conv_transpose2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_pad_replicate_negative_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_norm_inf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_rad2deg_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_renorm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_sign_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_squeeze_multiple_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_std_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_topk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_vstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_diagonal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_jiterator_binary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_mT_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_median_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_mm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_embedding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_grid_sample_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_mish_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_normal_number_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_polygamma_polygamma_n_2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_randint_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_randint_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_resize_as__cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_signal_windows_nuttall_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_slice_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_squeeze_multiple_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_stack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_trapezoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_unsafe_split_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_view_as_complex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops___rmatmul___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_addmv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_allclose_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_bitwise_not_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_cholesky_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_cumulative_trapezoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_empty_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_frexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_imag_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_isclose_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_jiterator_2inputs_2outputs_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_inv_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_vander_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_var_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_median_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_mm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_ne_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_avg_pool1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_conv_transpose2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_elu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_pad_reflect_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_rms_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_pinverse_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_rad2deg_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_resize_as__cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_sinc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_sinh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_erfcx_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_transpose_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_var_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_vstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_cuda_uint8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_tensor_overload_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_cuda_complex128, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_full_cuda_bool, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_full_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_cuda_int32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_cuda_int8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_tensor_overload_cuda_float16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_zeros_cuda_float64, test/test_ops.py::TestTagsCUDA::test_tags___rxor___cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_amax_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_cumsum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_empty_like_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_exp_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_expm1_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_ifftshift_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_frexp_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_gt_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_index_add_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_mean_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_mish_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_pairwise_distance_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_relu6_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_threshold_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_repeat_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_sigmoid_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_bessel_j1_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_where_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_cholesky_inverse_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_count_nonzero_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_empty_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_expm1_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_jiterator_binary_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_ldl_factor_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_matrix_power_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_mH_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_logsumexp_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_adaptive_max_pool2d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_cross_entropy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_embedding_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_relu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_softplus_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_normal_in_place_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_round_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_rsqrt_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_sgn_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_signal_windows_general_cosine_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_entr_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_scaled_modified_bessel_k1_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_std_mean_unbiased_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_stft_cuda_float32
2025-07-17T09:38:00.7091228Z 
2025-07-17T09:38:02.8527294Z 
2025-07-17T09:38:02.8528072Z test_ops 8/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_8.20_08973061562f9966_.log
2025-07-17T09:38:02.9020271Z Running 1644 items in this shard: test/test_ops.py::TestSelfKwarg::test_self_kwargs, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs__conversions_cfloat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs__conversions_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs__conversions_long_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_as_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_atleast_2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_flip_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_linspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_logaddexp2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_relu6_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_view_as_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_bitwise_right_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_compare_cpu_cauchy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_dist_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_double_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_index_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_index_reduce_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_isin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_ldl_factor_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_logcumsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_masked_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_new_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_cosine_similarity_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nonzero_static_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_norm_inf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_reshape_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_scatter_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_special_shifted_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_special_shifted_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_zero__cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_atleast_3d_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_bool_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_column_stack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_eq_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_mT_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_masked_fill_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_new_empty_strided_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_nn_functional_conv3d_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_nn_functional_conv_transpose2d_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_split_with_sizes_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_squeeze_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_atan_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_broadcast_to_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_cumsum_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_diagonal_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_erfinv_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_flatten_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_floor_divide_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_index_add_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_logical_and_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_pdist_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_sgn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_tril_indices_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_unsqueeze_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_vdot_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_allclose_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_argwhere_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_bincount_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_bitwise_or_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_bitwise_xor_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cdouble_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_column_stack_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_contiguous_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cumsum_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_dist_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_div_trunc_rounding_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_exponential_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_eye_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fft_ihfft_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_inner_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_item_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_cholesky_ex_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_slogdet_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_log_softmax_with_dtype_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_logical_or_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_argmax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_normalize_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_matmul_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_max_pool2d_with_indices_backward_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_meshgrid_list_of_tensors_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_mul_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_multinomial_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_adaptive_max_pool1d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_hardshrink_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_max_pool3d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_pixel_shuffle_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_relu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_soft_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nonzero_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_norm_fro_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_normal_in_place_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_outer_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_randint_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_round_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_rsqrt_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_sign_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_squeeze_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_tile_cuda, test/test_ops.py::TestCommonCUDA::test_errors_T_cuda, test/test_ops.py::TestCommonCUDA::test_errors_clamp_max_cuda, test/test_ops.py::TestCommonCUDA::test_errors_div_trunc_rounding_cuda, test/test_ops.py::TestCommonCUDA::test_errors_dsplit_cuda, test/test_ops.py::TestCommonCUDA::test_errors_fft_hfft2_cuda, test/test_ops.py::TestCommonCUDA::test_errors_heaviside_cuda, test/test_ops.py::TestCommonCUDA::test_errors_linalg_lstsq_grad_oriented_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_avg_pool3d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_signal_windows_general_hamming_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_sum_layout4_cuda, test/test_ops.py::TestCommonCUDA::test_errors_special_chebyshev_polynomial_w_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_erfinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_ihfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_floor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_gt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_slogdet_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_log1p_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_logical_not_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_mm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_nn_functional_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_signbit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_sort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_log_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_ndtri_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_topk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_true_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_all_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_asin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_asinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_baddbmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_bincount_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_bool_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_combinations_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cross_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cumulative_trapezoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cumulative_trapezoid_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_flip_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_floor_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_hypot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_index_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_item_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_lgamma_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_inv_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_matrix_rank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_tensorinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_fill_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_new_zeros_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_interpolate_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pad_reflect_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_relu_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_triplet_margin_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_normal_number_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_polygamma_polygamma_n_0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_repeat_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_signal_windows_cosine_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_modified_bessel_k0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tensor_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_triu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_view_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_vstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_T_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values___rdiv___cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_argsort_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_bool_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_chunk_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_diag_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_index_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_lgamma_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_logical_not_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_new_full_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_new_zeros_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_bessel_j0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_t_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_take_along_dim_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_transpose_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_unfold_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_zeros_cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addmv_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_angle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atan2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bitwise_or_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_broadcast_to_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_broadcast_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bucketize_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_byte_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cholesky_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_conj_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_constant_pad_nd_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_corrcoef_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cos_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_count_nonzero_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cross_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_double_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_dsplit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_erf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ifftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fliplr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_float_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fmod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_full_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_gcd_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_geometric_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_igamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_fill_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_eig_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_eigvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_pinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_pinv_hermitian_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logical_not_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mH_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_neg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_cosine_embedding_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_normalize_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_normalize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pixel_unshuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_relu6_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_selu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nonzero_static_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_normal_number_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_pinverse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sign_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_i1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_zeta_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_squeeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_std_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_svd_lowrank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_take_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_transpose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_view_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_vsplit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_zeros_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_zeros_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_numpy_ref_broadcast_to_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_broadcast_to_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_clone_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_linalg_cross_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_l1_loss_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_pairwise_distance_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_roll_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_signal_windows_kaiser_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_squeeze_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_tril_indices_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_view_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_out___rmul___cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_addcdiv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_clamp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_conj_physical_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_fftshift_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fliplr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fmod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_lerp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_linalg_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_masked_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_new_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_multigammaln_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_squeeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_addcdiv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_asin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_broadcast_shapes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cumprod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_div_floor_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_expand_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_exponential_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_frac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_gt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_imag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_linalg_eig_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_lu_unpack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_argmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_adaptive_max_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_normalize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_pad_replicate_negative_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_randint_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_ravel_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_angle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cumsum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_expm1_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_ifft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_lu_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_slogdet_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_lu_unpack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_mm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_mul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_permute_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_round_decimals_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sinc_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_slice_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_erfcx_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_squeeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_take_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_view_as_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning___rpow___cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_bool_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_add_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_amax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_diagonal_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_div_floor_rounding_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_div_trunc_rounding_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_expm1_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_ifftn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_floor_divide_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_item_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_linspace_tensor_overload_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_new_empty_strided_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_poisson_nll_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_sigmoid_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_spherical_bessel_j0_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_tril_indices_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_unsqueeze_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_amin_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_bernoulli_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_bitwise_and_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_fft2_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_fft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_rfft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fill_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_flip_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_hsplit_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_igammac_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_index_reduce_amin_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_cross_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_lu_factor_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_lu_solve_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_median_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_batch_norm_without_cudnn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_bilinear_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_cross_entropy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_ctc_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_gelu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_hardshrink_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_interpolate_linear_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nonzero_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_quantile_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_rot90_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_rsub_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_signal_windows_kaiser_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_split_list_args_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_split_with_sizes_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_svd_lowrank_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_tensor_split_cuda, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float___rdiv___cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atan2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_deg2rad_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erfc_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erfc_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erfinv_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_i0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log1p_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_masked_std_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_masked_std_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_mvlgamma_mvlgamma_p_3_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_mvlgamma_mvlgamma_p_5_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_rad2deg_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_rsqrt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sinc_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sinc_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_u_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_v_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_tan_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_true_divide_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bfloat16_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cdouble_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cfloat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cfloat_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cfloat_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_double_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_long_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_long_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acosh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_add_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcmul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_alias_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_alias_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_all_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amax_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amin_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_any_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_arange_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_partial_views_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_partial_views_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_1d_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_2d_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_3d_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_left_shift_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_left_shift_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_not_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_xor_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_to_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cauchy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_max_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_min_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_contiguous_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cos_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_scatter_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_digamma_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_digamma_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_floor_rounding_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_no_rounding_mode_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dsplit_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_like_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eq_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_equal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erfinv_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_as_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expm1_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eye_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfftn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flatten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flipud_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flipud_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_float_power_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmax_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ge_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ge_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_heaviside_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_heaviside_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_i0_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_add_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_fill_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isfinite_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isfinite_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isinf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isnan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isneginf_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isposinf_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_item_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_item_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_item_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_cross_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_cross_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_diagonal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_vector_norm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_vector_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logaddexp2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_masked_fill_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_variadic_tensors_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_minimum_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_minimum_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nan_to_num_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nan_to_num_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_native_layer_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ne_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_neg_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_zeros_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_alpha_dropout_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_celu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_l1_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_margin_ranking_loss_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_shuffle_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_unshuffle_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_unshuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_relu_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_selu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_normal_number_mean_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_positive_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rad2deg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rad2deg_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ravel_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ravel_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_real_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reciprocal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_roll_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_roll_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_round_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_select_scatter_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sigmoid_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sigmoid_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sign_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_signbit_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_signbit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_entr_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_entr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i0e_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_3_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_spherical_bessel_j0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_multiple_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_multiple_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sub_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_to_size_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_to_size_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_to_size_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_true_divide_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_diag_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_le_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_linspace_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_narrow_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_nn_functional_softshrink_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_T_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_byte_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_byte_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cdouble_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cfloat_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_char_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_char_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_char_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_double_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_float_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_int_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addr_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_alias_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_all_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_all_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_amin_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_amin_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_amin_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_any_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_arange_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_partial_views_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asinh_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_shapes_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cat_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_min_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clone_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_constant_pad_nd_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cos_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cosh_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cumsum_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_digamma_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dot_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_like_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_like_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_equal_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfc_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expm1_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eye_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftn_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfftn_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfftn_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flatten_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flipud_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmin_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_heaviside_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_heaviside_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_heaviside_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hsplit_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hsplit_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hstack_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hstack_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hstack_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_i0_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_select_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_select_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_select_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isinf_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isinf_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isposinf_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isposinf_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isposinf_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_istft_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_item_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_item_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lgamma_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_diagonal_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_matrix_norm_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_svd_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_tensor_overload_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log1p_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_normal_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_softmax_with_dtype_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logaddexp2_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logaddexp_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logsumexp_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_masked_fill_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_maximum_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_variadic_tensors_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_variadic_tensors_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_movedim_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_movedim_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_movedim_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_neg_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_full_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_ones_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_celu_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_log_softmax_with_dtype_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_log_softmax_with_dtype_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_margin_ranking_loss_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_unshuffle_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_prelu_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_relu6_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_selu_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmin_with_dtype_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_triplet_margin_loss_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_norm_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_remainder_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_as_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_as_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_round_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_round_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_select_scatter_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sin_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinc_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinh_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinh_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_softmax_with_dtype_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_bessel_j0_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_entr_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_entr_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1e_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_softmax_with_dtype_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_softmax_with_dtype_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_logit_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_3_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtr_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtri_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_softmax_with_dtype_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_softmax_with_dtype_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_spherical_bessel_j0_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_square_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_copy_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_std_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sub_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_copy_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tan_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tanh_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_copy_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tril_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trunc_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_copy_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_var_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_var_mean_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vdot_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_where_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_xlogy_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bfloat16_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bfloat16_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bfloat16_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bool_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cdouble_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_complex_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_double_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_long_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_long_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acosh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_partial_views_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_partial_views_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_scatter_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_3d_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_or_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_or_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bucketize_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cat_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_chunk_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_max_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_max_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_min_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cos_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cosh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_count_nonzero_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumsum_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_embed_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_embed_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_digamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_trunc_rounding_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dsplit_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_like_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eq_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfc_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftshift_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftshift_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftshift_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfftn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftshift_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fliplr_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmin_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_geometric_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_geometric_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gt_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gt_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_i0_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_imag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_select_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isinf_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isinf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isnan_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isposinf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isposinf_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_le_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lgamma_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_diagonal_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_diagonal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log1p_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_not_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logsumexp_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lt_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_variadic_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_variadic_tensors_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_minimum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_movedim_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nan_to_num_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_neg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_ones_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_gelu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hardtanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_margin_ranking_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pdist_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_unshuffle_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_relu6_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_relu_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softplus_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_triplet_margin_loss_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_normal__in_place_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_normal_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_positive_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_positive_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_pow_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_prod_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_real_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reciprocal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_remainder_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_renorm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_repeat_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsub_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_select_scatter_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sign_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sign_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_signbit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinc_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_entr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_erfcx_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i0e_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i0e_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_logit_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_5_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_5_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sqrt_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_square_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sub_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sub_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_to_size_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tan_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trace_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_triu_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_triu_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vdot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vstack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_zeros_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bfloat16_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_chalf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_char_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_double_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_double_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_half_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_half_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_half_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_int_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_long_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_polar_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcmul_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcmul_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addr_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amin_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amin_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amin_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_arange_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_partial_views_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asin_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asinh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asinh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asinh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_not_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_right_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_block_diag_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cat_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ceil_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_chunk_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_min_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_min_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_column_stack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_column_stack_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cos_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cos_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cosh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cosh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_embed_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_scatter_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_trunc_rounding_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dsplit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_as_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expm1_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exponential_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eye_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft2_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifftshift_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flatten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flip_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flip_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fliplr_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flipud_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flipud_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmax_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_gt_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hsplit_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_add_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_select_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isfinite_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isnan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_matrix_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log10_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log10_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log1p_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logaddexp2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_and_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_or_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logspace_tensor_overload_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logspace_tensor_overload_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_masked_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_variadic_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_minimum_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_movedim_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_neg_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_strided_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nextafter_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nextafter_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_alpha_dropout_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_celu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_channel_shuffle_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_glu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_unshuffle_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_prelu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_relu6_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_tanhshrink_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_tanhshrink_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_threshold_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_threshold_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_triplet_margin_loss_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ravel_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ravel_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reciprocal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_renorm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_roll_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rot90_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rot90_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_signbit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinc_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_bessel_j1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_entr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_erfcx_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1e_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_1_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_5_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_zeta_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sub_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sub_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_take_along_dim_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_triu_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_triu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vdot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vstack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_xlogy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager__unsafe_masked_index_put_accumulate_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_aminmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_argwhere_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_atleast_1d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_atleast_2d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_block_diag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_broadcast_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_broadcast_to_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_bucketize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_char_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_clamp_max_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_column_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_combinations_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_conj_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_corrcoef_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cummax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cumulative_trapezoid_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_deg2rad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_diagflat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_irfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_irfftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_index_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_istft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_cond_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logical_not_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logsumexp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_mean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_matmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nanmedian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nansum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_conv1d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_interpolate_nearest_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_logsigmoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_softsign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_norm_inf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_ones_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_qr_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_renorm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_round_decimals_neg_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_signal_windows_nuttall_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_transpose_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_transpose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unfold_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unsafe_chunk_cuda_complex64, test/test_ops.py::TestCompositeComplianceCUDA::test_backward___getitem___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_amax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_amin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_bfloat16_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cholesky_inverse_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cummax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_erf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_exp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_expand_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_fftshift_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_cholesky_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_softsign_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_reciprocal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_slice_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_std_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_unbind_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_unfold_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input___rmod___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_addmv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_diagonal_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_expm1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_rfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_index_reduce_amax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_det_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_vector_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_lu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_movedim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nanmean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_native_batch_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_adaptive_avg_pool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_conv_transpose3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_cosine_similarity_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_gaussian_nll_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_pad_reflect_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_selu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_qr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_resize_as__cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_scalar_tensor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_split_list_args_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_triangular_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_tril_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_unbind_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_var_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_T_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_addmm_decomposed_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_atleast_1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_bfloat16_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_einsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_half_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_inner_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_le_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_cholesky_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_diagonal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_lu_factor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_logsumexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_select_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_multinomial_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_narrow_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nextafter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_linear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_rms_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_outer_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_quantile_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_real_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_searchsorted_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_signal_windows_hann_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sparse_mm_reduce_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_split_list_args_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_std_mean_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_unique_consecutive_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_vdot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_view_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_zeros_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_amax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_corrcoef_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_cummin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_geometric_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_hstack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_index_reduce_amax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_index_reduce_amin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_lu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_fill_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_native_dropout_backward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_hardsigmoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_pca_lowrank_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_reshape_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_round_decimals_0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_select_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_signal_windows_nuttall_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_slice_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_zeta_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_tanh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_asinh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_chunk_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_clamp_min_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_empty_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_exponential_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_ihfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_ge_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_int_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_le_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_lerp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_new_empty_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_embedding_bag_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_pca_lowrank_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_resize_as__cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_round_decimals_neg_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_scatter_reduce_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_scaled_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_xlog1py_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_svd_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_unsafe_split_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_xlogy_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view___rpow___cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_as_strided_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_asin_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_cumprod_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_ifft2_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_irfftn_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_isfinite_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_linspace_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_mean_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_renorm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_view_as_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_where_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_addmm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_as_strided_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_baddbmm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cartesian_prod_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cos_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_diag_embed_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_float_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_index_add_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_lu_factor_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_matrix_rank_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linspace_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_ones_like_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_rot90_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_rsqrt_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_scalar_tensor_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_std_unbiased_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_svd_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_t_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_uniform_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_zero__cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_as_strided_partial_views_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_item_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_linspace_tensor_overload_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_logspace_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_movedim_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_rsqrt_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_special_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_triu_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_vsplit_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__unsafe_masked_index_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_atleast_3d_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fft_fftn_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fft_ifftn_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_hstack_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_cross_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_eig_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_lu_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_masked_scatter_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_softmin_with_dtype_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_triplet_margin_with_distance_loss_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_roll_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_uniform_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_view_as_real_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view___rdiv___cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view___rmul___cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs__conversions_half_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_cauchy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fft_hfftn_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fft_ifft_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fft_ihfft_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fill_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_linspace_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_logaddexp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_logspace_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_lt_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_ne_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_hardshrink_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_prelu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_relu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_prod_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_std_mean_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_sub_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_unbind_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_addcdiv_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_as_strided_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_asinh_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_bernoulli_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_diagflat_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_dot_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_eq_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_erfc_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_index_reduce_amin_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_isnan_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_isreal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_cross_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_lt_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_log_softmax_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_var_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_new_empty_strided_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_adaptive_avg_pool3d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_conv_transpose3d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_dropout2d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_hardtanh_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_huber_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_softsign_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_outer_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signal_windows_hamming_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signbit_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_sort_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_airy_ai_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_chebyshev_polynomial_w_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_spherical_bessel_j0_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_sqrt_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_std_mean_unbiased_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_t_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_tanh_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_xlogy_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake___rxor___cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_add_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_asin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast___rsub___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_addmm_decomposed_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_baddbmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_conj_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cumulative_trapezoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_digamma_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_div_trunc_rounding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_exp2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_full_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_imag_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_isnan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_svdvals_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_log1p_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_msort_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_adaptive_max_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_conv_transpose2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_cosine_similarity_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_interpolate_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_pca_lowrank_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signal_windows_exponential_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_modified_bessel_i1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_topk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_tril_indices_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_unsafe_split_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_zeros_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_byte_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_clone_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_conj_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_cumulative_trapezoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_ihfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_eigh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_qr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_svd_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_vander_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_masked_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_mode_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nansum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_embedding_bag_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_pairwise_distance_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_torch_ops_aten__efficient_attention_forward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp___rmod___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp__batch_norm_with_update_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_chunk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cumulative_trapezoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_dstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_expand_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_flatten_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_index_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_inner_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_kthvalue_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_log_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nanmedian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_cross_entropy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_elu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_gaussian_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_max_unpool2d_grad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_pad_reflect_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_ravel_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_squeeze_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_sub_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_tensordot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_triu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_var_mean_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_diag_embed_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_diagonal_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_einsum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_empty_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_ifftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_gcd_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_istft_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_fake_lgamma_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_log_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logical_xor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_logaddexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_softmin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_new_ones_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_norm_fro_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_short_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_i1e_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_std_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_svd_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_unfold_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_uniform_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_zeros_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops___getitem___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops___rxor___cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_addcdiv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_atleast_2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_broadcast_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_diagflat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_expm1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_hfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_gcd_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_histc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_i0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_jiterator_binary_return_by_ref_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_kthvalue_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_le_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_diagonal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_eigh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_matrix_power_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_matrix_rank_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_log_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_lu_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_lu_unpack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_sum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_normalize_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_pad_replicate_negative_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_silu_complex_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_softplus_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_renorm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_repeat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_rsub_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_sgn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_signal_windows_blackman_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_modified_bessel_k0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_sum_to_size_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_unflatten_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_xlogy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_ones_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_arange_cuda_int32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_cuda_complex128, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_tensor_overload_cuda_bfloat16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_tensor_overload_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_tensor_overload_cuda_uint8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_tensor_overload_cuda_complex128, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_ones_cuda_int32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_ones_cuda_int8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_zeros_cuda_bfloat16, test/test_ops.py::TestTagsCUDA::test_tags_H_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__chunk_cat_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_T_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs__conversions_chalf_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_abs_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_bitwise_xor_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_bucketize_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_hfft_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_irfftn_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_gcd_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_isposinf_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_logspace_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_minimum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_normal__in_place_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_ones_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_sin_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_i0e_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_unbind_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_unfold_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_vstack_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_alias_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_bitwise_and_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags_broadcast_shapes_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_clamp_min_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_contiguous_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_diagonal_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_einsum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_ifftn_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_ihfft_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_float_power_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_frac_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_igammac_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_index_reduce_amin_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_isinf_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_svdvals_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_log10_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_cumprod_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_cumsum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_narrow_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_relu6_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_sign_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_slice_scatter_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_sparse_mm_reduce_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_ndtri_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_squeeze_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_topk_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_trapezoid_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_uniform_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_var_mean_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_where_cuda_float32
2025-07-17T09:38:02.9447657Z 
2025-07-17T09:38:48.8377506Z 
2025-07-17T09:38:48.8378514Z test_ops 11/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_11.20_10d8557352556eab_.log
2025-07-17T09:38:48.8988658Z Running 1743 items in this shard: test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_as_strided_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_atan2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_bucketize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_diagonal_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_index_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_new_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_reshape_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_special_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_split_with_sizes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_unsqueeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_addcdiv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_addcmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_atan2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_cov_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_expand_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_igamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_index_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_eigvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_logsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_cross_entropy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_kl_div_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_max_unpool2d_grad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_rot90_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_special_laguerre_polynomial_l_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_sum_to_size_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_trapezoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_tril_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_true_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_uniform_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_vdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_as_strided_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_double_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_dstack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_empty_like_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_fft_hfft_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_fft_ifftshift_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_full_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_unsqueeze_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes___rsub___cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs__conversions_float_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_acos_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_addr_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_clamp_max_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_div_floor_rounding_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_empty_strided_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_floor_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_gcd_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_index_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_log2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_log_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_log_normal_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_huber_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_triplet_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_ones_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_reshape_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_sinh_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_special_bessel_j1_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_squeeze_multiple_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_stack_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_tanh_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_transpose_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_acosh_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_as_strided_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_bfloat16_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_bitwise_right_shift_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cdist_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_chalf_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cholesky_inverse_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_clamp_min_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cumulative_trapezoid_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_digamma_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_empty_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_flatten_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fliplr_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_float_power_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_lerp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_cross_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_lu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_svdvals_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_logical_and_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_logical_not_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_long_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_lu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_max_reduction_no_dim_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_mvlgamma_mvlgamma_p_5_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_new_ones_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_gaussian_nll_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_normal_number_mean_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_rad2deg_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_real_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_roll_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_rot90_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_scatter_add_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_scatter_reduce_sum_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_shifted_chebyshev_polynomial_v_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_sub_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_tanh_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_transpose_copy_cuda, test/test_ops.py::TestCommonCUDA::test_errors_amax_cuda, test/test_ops.py::TestCommonCUDA::test_errors_cov_cuda, test/test_ops.py::TestCommonCUDA::test_errors_div_no_rounding_mode_cuda, test/test_ops.py::TestCommonCUDA::test_errors_ldexp_cuda, test/test_ops.py::TestCommonCUDA::test_errors_linspace_tensor_overload_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_adaptive_avg_pool1d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_max_pool1d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_multi_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_errors_signal_windows_cosine_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_mul_layout4_cuda, test/test_ops.py::TestCommonCUDA::test_errors_special_hermite_polynomial_h_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sub_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_cholesky_inverse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_rfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_inv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_lstsq_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_logical_or_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_logspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_neg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_qr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_erfcx_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_shifted_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_zeta_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_sqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_where_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices___rdiv___cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_amin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_atleast_1d_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_atleast_3d_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_block_diag_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_broadcast_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_broadcast_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_broadcast_to_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_ceil_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_corrcoef_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cummax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_diag_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_double_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_dsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_empty_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_empty_strided_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_expand_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_hfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_ifft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_flipud_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_floor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_index_reduce_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_isneginf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_isreal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_ldexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_solve_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_vector_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_log2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logical_xor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_avg_pool1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_ctc_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_elu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_feature_alpha_dropout_without_train_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pad_reflect_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nonzero_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_norm_fro_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_polygamma_polygamma_n_4_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_repeat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_repeat_interleave_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_reshape_as_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_select_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sigmoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sigmoid_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_signal_windows_exponential_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_signal_windows_hamming_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_signal_windows_nuttall_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tensor_split_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_transpose_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tril_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unbind_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unique_consecutive_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_vsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_abs_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_atleast_2d_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_bitwise_xor_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_block_diag_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_broadcast_to_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_clone_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_diagonal_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_empty_like_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fliplr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_masked_select_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_nn_functional_pixel_unshuffle_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_nonzero_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_polygamma_polygamma_n_0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_prod_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_repeat_interleave_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_rot90_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_sin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_squeeze_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_stack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_true_divide_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_view_as_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_zero__cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___rsub___cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_argmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bitwise_and_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bucketize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ceil_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ceil_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_conj_physical_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_constant_pad_nd_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_contiguous_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_contiguous_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cosh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_count_nonzero_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diff_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_expand_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_expand_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_eye_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_hfft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ifft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ifftshift_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_irfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_irfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_rfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_float_power_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_heaviside_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_hstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_reduce_amax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_eigh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_eigvals_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lstsq_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_norm_subgradients_at_zero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_tensorsolve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_vander_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linspace_tensor_overload_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mH_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_logsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_matmul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_minimum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mvlgamma_mvlgamma_p_5_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_narrow_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_new_ones_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_adaptive_max_pool1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_adaptive_max_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_conv_transpose3d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_ctc_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_gaussian_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_linear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_poisson_nll_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_tanhshrink_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_positive_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_put_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_reciprocal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_reshape_as_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_rot90_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_round_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sgn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sinc_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_modified_bessel_k0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_ndtr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_shifted_chebyshev_polynomial_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_t_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_trapezoid_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_var_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_where_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_numpy_ref_allclose_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_equal_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_flatten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_roll_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_squeeze_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_out__refs_asinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_column_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_count_nonzero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_dstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_equal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_irfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_isneginf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_sin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_i0e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_spherical_bessel_j0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_zeta_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_transpose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_tril_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_any_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_bitwise_right_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out_broadcast_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_combinations_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_diff_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_empty_permuted_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_float_power_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_geqrf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_heaviside_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_histc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_hstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_le_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_cholesky_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_logit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_logspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_minimum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_cosine_similarity_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_interpolate_nearest-exact_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_permute_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_asin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_diag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_exp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_fft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_hfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_ifft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_ifftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_eigvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_norm_subgradients_at_zero_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nanmean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_norm_fro_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_ones_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_ormqr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_outer_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_polygamma_polygamma_n_0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_pow_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_qr_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_reciprocal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_remainder_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sin_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_special_erfcx_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_std_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_svd_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_tan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_topk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_tril_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_true_divide_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_unfold_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_unsqueeze_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_vstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_vstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_signal_windows_kaiser_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_shifted_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_std_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_topk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_unfold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_var_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_var_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning___rmatmul___cuda, test/test_ops.py::TestCommonCUDA::test_out_warning___ror___cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__batch_norm_with_update_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__native_batch_norm_legit_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_cdouble_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_chalf_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_abs_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_arange_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_broadcast_tensors_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_exp2_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_irfft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_hstack_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_igamma_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_isfinite_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_isneginf_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_lcm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_linalg_matrix_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_meshgrid_variadic_tensors_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_leaky_relu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_rot90_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_select_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_entr_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_unfold_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_view_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_where_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__segment_reduce_lengths_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_asinh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_combinations_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_complex_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_diag_embed_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_diagonal_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_div_no_rounding_mode_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_ihfft2_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fmax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_full_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_grid_sampler_2d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_histogramdd_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_hypot_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_jiterator_unary_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_inv_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_tensorinv_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_vander_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_logdet_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_minimum_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_celu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_margin_ranking_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_prelu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_polygamma_polygamma_n_1_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_scalar_tensor_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_scatter_add_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_searchsorted_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_bessel_y1_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_scaled_modified_bessel_k1_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_std_mean_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_topk_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_vdot_cuda, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_acos_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atan_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_deg2rad_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erf_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_expm1_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_i0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_mvlgamma_mvlgamma_p_1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_rad2deg_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_v_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_xlog1py_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_xlog1py_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_tan_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_T_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_T_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bfloat16_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_byte_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_char_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcmul_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcmul_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_alias_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_allclose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_partial_views_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asinh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_and_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_or_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_block_diag_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_tensors_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_to_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_to_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cat_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cauchy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ceil_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_chunk_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_min_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_contiguous_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_contiguous_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_copysign_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_copysign_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cosh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_embed_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_embed_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_digamma_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_no_rounding_mode_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eq_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exponential_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eye_cuda_float8_e4m3fn, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eye_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfftn_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fill_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fill_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flipud_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_float_power_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmax_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmin_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_frexp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isinf_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isnan_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_svdvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_vecdot_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log1p_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_and_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_not_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_or_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_maximum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_maximum_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_variadic_tensors_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_movedim_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_neg_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_strided_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_zeros_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_zeros_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_channel_shuffle_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_elu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pairwise_distance_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_shuffle_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_poisson_nll_loss_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_relu6_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_selu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_tanhshrink_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_triplet_margin_loss_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_normal_number_mean_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rad2deg_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_randn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_real_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_repeat_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_as_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_as_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_round_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinc_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_bessel_j0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_entr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i1e_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_ndtr_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_ndtr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_ndtri_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_spherical_bessel_j0_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_std_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sub_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_take_along_dim_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_take_along_dim_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tan_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_to_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_to_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_triu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_triu_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_triu_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trunc_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_var_mean_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_as_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_where_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_div_floor_rounding_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_fmin_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_geometric_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_linalg_cross_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_minimum_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_special_zeta_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_T_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_T_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cdouble_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_half_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_int_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_int_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_long_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_short_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcdiv_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcmul_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addr_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_amin_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_any_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_copy_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_3d_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_3d_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_xor_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_tensors_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_tensors_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_to_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bucketize_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_column_stack_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_constant_pad_nd_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_contiguous_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_contiguous_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_copysign_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cos_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cumprod_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cumprod_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cumsum_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_deg2rad_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_scatter_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_scatter_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_digamma_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_trunc_rounding_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_trunc_rounding_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dot_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dsplit_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dstack_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_equal_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erf_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfinv_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfinv_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eye_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eye_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft2_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftn_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftn_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftshift_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftshift_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftshift_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flatten_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmax_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ge_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_gt_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isinf_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_istft_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_item_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lerp_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lgamma_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_cross_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_diagonal_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_softmax_with_dtype_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_softmax_with_dtype_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_and_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logspace_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_meshgrid_variadic_tensors_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_copy_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_full_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_celu_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_dropout_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_elu_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_gelu_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_group_norm_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_log_softmax_with_dtype_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_log_softmax_with_dtype_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pairwise_distance_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_shuffle_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_unshuffle_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_unshuffle_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_poisson_nll_loss_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_relu_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_relu_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmin_with_dtype_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softshrink_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_triplet_margin_loss_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_norm_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_norm_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ones_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rad2deg_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_real_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_as_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_round_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsqrt_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sigmoid_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sign_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sign_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sin_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinc_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_erfcx_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i0e_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_1_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_5_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtri_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_softmax_with_dtype_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_spherical_bessel_j0_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_stack_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_stack_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_stack_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_std_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_std_mean_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sub_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sub_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_to_size_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tan_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tensor_split_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tensor_split_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_copy_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_true_divide_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trunc_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_copy_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_copy_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_copy_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vsplit_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_zeros_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bfloat16_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bfloat16_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bool_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bool_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_complex_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_double_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acosh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acosh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcmul_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addr_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_amin_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_any_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_arange_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_arange_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_partial_views_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asin_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asinh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asinh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_1d_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_1d_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_and_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_left_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_to_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ceil_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_max_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clone_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_column_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cos_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cosh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumprod_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumprod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumprod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumprod_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_deg2rad_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_embed_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_embed_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_floor_rounding_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dsplit_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_strided_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_strided_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_equal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfc_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfinv_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfinv_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_as_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_as_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expm1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flip_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fliplr_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_float_power_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmax_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmod_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_i0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_imag_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isclose_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isinf_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isnan_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isneginf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_item_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lerp_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lgamma_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_cross_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_norm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_svdvals_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_vecdot_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_tensor_overload_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log1p_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_normal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_normal_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_xor_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_maximum_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mean_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_minimum_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_movedim_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_movedim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mul_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_native_layer_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_native_layer_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_neg_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_strided_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_full_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hardshrink_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_mse_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pairwise_distance_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pairwise_distance_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_shuffle_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_triplet_margin_loss_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ones_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_positive_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_positive_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_pow_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_pow_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_prod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rad2deg_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reciprocal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_renorm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_repeat_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sin_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_bessel_j0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_entr_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i0e_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_ndtr_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_3_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_5_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtr_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtr_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_square_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_to_size_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tan_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tan_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_true_divide_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unflatten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_var_mean_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vstack_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vstack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_where_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_where_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_xlogy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bfloat16_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bool_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bool_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_byte_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_byte_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_char_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_half_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_int_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_long_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_short_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_add_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcmul_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_alias_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_alias_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_alias_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_allclose_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amin_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_arange_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_partial_views_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asinh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_or_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_xor_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_xor_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ceil_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_chunk_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_min_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_physical_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_contiguous_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_copysign_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_copysign_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cos_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cos_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cos_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_scatter_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_scatter_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_digamma_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_floor_rounding_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dstack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_as_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_as_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expm1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eye_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifftshift_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft2_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flip_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flipud_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flipud_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_float_power_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_divide_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmod_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_gt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_heaviside_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_add_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_fill_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_select_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isclose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isfinite_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lgamma_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_cross_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_vecdot_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log1p_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_or_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_masked_fill_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_masked_fill_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_maximum_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_mean_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_movedim_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ne_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ne_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_neg_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_full_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_zeros_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_alpha_dropout_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_log_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_margin_ranking_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_margin_ranking_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_mish_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_shuffle_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_unshuffle_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_relu6_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_selu_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_smooth_l1_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmax_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softplus_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_pow_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rad2deg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rad2deg_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reciprocal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_remainder_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_renorm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_roll_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_roll_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rot90_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rot90_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_select_scatter_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sgn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sin_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_bessel_j0_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_bessel_j1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtr_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_spherical_bessel_j0_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_split_with_sizes_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_multiple_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_stack_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_std_mean_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sub_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tanh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_to_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_to_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trace_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trace_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tril_indices_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unflatten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_var_mean_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_var_mean_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vdot_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vsplit_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vsplit_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vstack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_zeros_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_T_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___radd___cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager__batch_norm_with_update_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_abs_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_bmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cholesky_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_column_stack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_expand_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_eye_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_ifftshift_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_float_power_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_geometric_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_hstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_index_reduce_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_int_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_isclose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_ldexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_householder_product_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_ldl_factor_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_matrix_power_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_norm_subgradients_at_zero_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_pinv_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_normalize_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_prod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_var_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_meshgrid_list_of_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_new_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_channel_shuffle_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_interpolate_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pixel_unshuffle_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_triplet_margin_loss_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_normal_in_place_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_prod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_randn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_repeat_interleave_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_reshape_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_resolve_conj_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sgn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sparse_mm_reduce_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_scaled_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_squeeze_multiple_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_std_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_to_sparse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_trapz_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_triu_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unsqueeze_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_vdot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_view_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_xlogy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_zero__cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward__softmax_backward_data_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_alias_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_as_strided_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_atan_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cfloat_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_conj_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_erfc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_exp2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_fft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_hfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_rfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_flipud_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_gather_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_index_reduce_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_log1p_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_native_batch_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_relu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_ormqr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_quantile_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_scatter_reduce_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_squeeze_multiple_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_unbind_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_unsafe_split_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_var_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_view_as_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input__softmax_backward_data_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_all_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_cartesian_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_cdouble_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_clamp_min_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_diagflat_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_einsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_equal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_ifft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_ihfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_inner_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_jiterator_4inputs_with_extra_args_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_jiterator_binary_return_by_ref_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_lgamma_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_eigvalsh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_tensorinv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linspace_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_log1p_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_maximum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_mode_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_new_empty_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_avg_pool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_norm_inf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_permute_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_real_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_laguerre_polynomial_l_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_tanh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad__softmax_backward_data_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_argmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_bmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_conj_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_diag_embed_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_div_floor_rounding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_empty_permuted_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_expand_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_ifftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_full_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_index_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_inv_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_logdet_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_matmul_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_new_ones_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_conv3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_ctc_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_dropout_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_mish_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_unfold_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_resize__cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_roll_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_short_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sigmoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sign_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_bessel_y1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_log_ndtr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_xlog1py_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_squeeze_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_var_mean_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator___rsub___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_abs_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_acos_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_argwhere_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_asin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_deg2rad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_diagonal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_empty_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_full_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_le_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_logical_or_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_mm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_native_layer_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nextafter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_conv_transpose1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_polygamma_polygamma_n_0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_rand_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_short_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_sinc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_xlog1py_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_take_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_triangular_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_unbind_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_unflatten_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_unsqueeze_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_view_as_complex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay___rmod___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_addcdiv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_addr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_cfloat_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_clone_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_digamma_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_ifft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_frexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_index_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_index_put_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_ldl_factor_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_vector_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_mT_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_rrelu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nonzero_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_norm_nuc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_permute_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_resolve_conj_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_rot90_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_split_with_sizes_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_std_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_t_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_unflatten_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_bfloat16_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_char_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_fft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_hsplit_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_index_add_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_nn_functional_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_nn_functional_triplet_margin_loss_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_stft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_sub_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_any_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_broadcast_tensors_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cfloat_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_chalf_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_constant_pad_nd_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_hsplit_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_triplet_margin_with_distance_loss_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_put_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_resize_as__cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_sparse_sampled_addmm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_std_mean_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_T_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs__conversions_chalf_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_abs_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_eq_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fill_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_index_select_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_isclose_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_isinf_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_log2_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_new_empty_strided_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_new_zeros_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_split_with_sizes_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_unbind_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_acosh_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_as_strided_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_atanh_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fft_ifftshift_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_matrix_rank_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_matrix_rank_hermitian_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_logcumsumexp_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nanmean_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_new_empty_strided_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_unfold_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_normal_in_place_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_scatter_add_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_sum_to_size_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_view_as_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_view_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view___getitem___cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view___rmod___cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_acosh_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_chunk_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_clone_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_deg2rad_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fft_rfft_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_index_select_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_isclose_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_le_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_linalg_matrix_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_linalg_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_mul_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_narrow_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_l1_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_pixel_unshuffle_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_ones_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_reshape_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_special_logit_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_trace_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_unsqueeze_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__unsafe_masked_index_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_acosh_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_add_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_atanh_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_ceil_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_complex_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cumsum_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_diag_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_div_no_rounding_mode_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fft_ifft_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fliplr_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_grid_sampler_2d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_i0_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_jiterator_4inputs_with_extra_args_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_lgamma_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_eigvalsh_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_ldl_solve_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_tensorinv_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_vander_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linspace_tensor_overload_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_lu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_lu_solve_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_median_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_movedim_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_conv1d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_interpolate_nearest_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_max_unpool1d_grad_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_rrelu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_scaled_dot_product_attention_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_softplus_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_triplet_margin_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_norm_nuc_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_positive_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_select_scatter_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_signal_windows_nuttall_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_modified_bessel_i0_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_modified_bessel_k0_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_shifted_chebyshev_polynomial_u_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_xlog1py_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_sub_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_tensordot_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_trace_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_zeros_like_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake___rmul___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake___rsub___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_acos_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_addbmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_atleast_2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_addcdiv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_bitwise_not_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_bitwise_right_shift_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_clamp_min_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_clone_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_diagonal_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_double_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_dstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_expand_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_ihfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_hstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_i0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_index_reduce_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_jiterator_binary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_lgamma_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_lu_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_logsumexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nan_to_num_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_conv_transpose3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_elu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_embedding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_silu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_soft_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_ormqr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_polygamma_polygamma_n_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signal_windows_general_cosine_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signal_windows_hamming_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signal_windows_hann_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_sparse_mm_reduce_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_airy_ai_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_bessel_j1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_shifted_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_take_along_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_var_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_conj_physical_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_addr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_cholesky_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_ifftshift_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_flatten_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_min_binary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_native_dropout_backward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_max_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_selu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_silu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_polar_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_qr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_rsub_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_scatter_reduce_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_take_along_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_var_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_view_as_complex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_zero__cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_acos_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cdouble_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_contiguous_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_hstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_index_reduce_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_logsumexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_neg_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_relu6_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_relu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_soft_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_pca_lowrank_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_pinverse_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_special_ndtri_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_std_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_var_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_irfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_flipud_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_frac_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_geqrf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_index_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_index_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_inner_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_le_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_vecdot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logspace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_mv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_native_batch_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_fractional_max_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_interpolate_nearest-exact_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_linear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_unfold_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_polygamma_polygamma_n_0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_randn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_signal_windows_general_hamming_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_softmax_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_modified_bessel_k0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_std_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_stft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_take_along_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_tan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops___rpow___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_all_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_argmin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_atan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_atleast_1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_bernoulli_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_block_diag_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_byte_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_contiguous_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_dist_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_flatten_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_float_power_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_hstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_index_reduce_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_inner_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_int_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_ldl_factor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_cumsum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_native_batch_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_new_ones_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_prelu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_threshold_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nonzero_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_ones_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_roll_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_round_decimals_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_scatter_reduce_sum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_squeeze_multiple_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_topk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_trace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_unsqueeze_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_zeros_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_zeros_cuda_complex128, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_full_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_tensor_overload_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_uint8, test/test_ops.py::TestTagsCUDA::test_tags__refs__conversions_bool_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_allclose_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_bitwise_left_shift_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_dstack_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_ihfftn_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fmax_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_lerp_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_linalg_diagonal_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_linalg_svd_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_linalg_vector_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_new_ones_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nextafter_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_sinc_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_xlog1py_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_squeeze_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_squeeze_multiple_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_std_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_tril_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_triu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_trunc_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_add_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_bernoulli_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_bitwise_right_shift_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags_ceil_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_cfloat_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_equal_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_exp2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_full_like_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_hsplit_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_householder_product_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_pinv_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_solve_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_native_layer_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_conv3d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_max_pool1d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_silu_complex_cuda_complex64, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_laguerre_polynomial_l_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_xlog1py_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_square_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_std_unbiased_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_unsqueeze_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_var_mean_unbiased_cuda_float32
2025-07-17T09:38:48.9440811Z 
2025-07-17T09:39:23.6090628Z 
2025-07-17T09:39:23.6091445Z test_ops 15/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_15.20_485bed552439e1ae_.log
2025-07-17T09:39:23.6590336Z Running 1686 items in this shard: test/test_ops.py::TestCommonCUDA::test_compare_cpu___rdiv___cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs__conversions_char_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs__conversions_short_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_arange_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_cauchy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_cumprod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_eye_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_index_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_normal__in_place_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_normal_number_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_special_zeta_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_stft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_addmv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_addr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_alias_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_bucketize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_constant_pad_nd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_half_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_index_reduce_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_inner_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_matrix_power_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_norm_subgradients_at_zero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_qr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_lu_unpack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_mT_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_mm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nanquantile_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_gaussian_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_relu6_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_trace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_unsafe_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_var_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_view_as_real_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_angle_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_fill_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_flatten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_float_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_isinf_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_randn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_sum_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_unflatten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_as_strided_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_conj_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_diag_embed_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_dstack_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_ifft_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fill_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_isfinite_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_group_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_normal_number_mean_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_ravel_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_sub_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_t_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_trunc_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_unfold_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_view_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_byte_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cat_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cauchy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_copysign_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cov_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_cumprod_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_diag_embed_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fft_ihfftn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_gt_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_index_reduce_prod_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_isinf_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_isreal_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_matrix_rank_hermitian_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_solve_triangular_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_logsumexp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_fill_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_bilinear_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_conv3d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_gelu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_glu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_interpolate_nearest-exact_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_l1_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_multilabel_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_threshold_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_resolve_neg_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_round_decimals_3_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_chebyshev_polynomial_u_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_legendre_polynomial_p_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_scaled_modified_bessel_k1_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_std_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_tensordot_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_unique_consecutive_cuda, test/test_ops.py::TestCommonCUDA::test_errors___rpow___cuda, test/test_ops.py::TestCommonCUDA::test_errors_atan2_cuda, test/test_ops.py::TestCommonCUDA::test_errors_bitwise_right_shift_cuda, test/test_ops.py::TestCommonCUDA::test_errors_le_cuda, test/test_ops.py::TestCommonCUDA::test_errors_min_binary_cuda, test/test_ops.py::TestCommonCUDA::test_errors_movedim_cuda, test/test_ops.py::TestCommonCUDA::test_errors_ne_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_embedding_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_group_norm_cuda, test/test_ops.py::TestCommonCUDA::test_errors_signal_windows_gaussian_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_argmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_clamp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_erf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_ifft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_kthvalue_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_multi_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_vecdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_log10_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_logsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_lu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_scatter_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_sin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_scaled_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_unfold_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_xlogy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_T_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices___rmul___cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_diagonal_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_expand_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_fft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_fftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_rfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_full_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_gt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_histc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_index_put_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_jiterator_binary_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_jiterator_unary_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_ldl_factor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_log1p_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mH_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_normalize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nanmedian_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_new_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_new_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_new_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_cosine_similarity_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_max_unpool2d_grad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_softmin_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_softsign_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_tanhshrink_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nonzero_static_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_polygamma_polygamma_n_1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_put_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_real_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_roll_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_rot90_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_rot90_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sign_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_signal_windows_general_cosine_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_slice_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_bessel_j0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_bessel_y0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_split_list_args_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_std_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_std_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sub_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_transpose_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unique_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_view_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_xlogy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values___rmul___cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_atan_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_bfloat16_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_bitwise_and_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_constant_pad_nd_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_dsplit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_eye_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fft_rfftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_jiterator_binary_return_by_ref_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_max_reduction_no_dim_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_repeat_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_reshape_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_scatter_add_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_short_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_chebyshev_polynomial_v_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_laguerre_polynomial_l_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_ndtri_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_unbind_cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___getitem___cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___radd___cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___rxor___cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addbmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addr_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_all_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_any_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_partial_views_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_baddbmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_chalf_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_chunk_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_clamp_max_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cov_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diagonal_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_double_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_empty_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_empty_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_eq_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_expand_as_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_eye_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_hfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ihfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_rfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_flipud_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_flipud_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_full_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_imag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_add_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ldexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_cholesky_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_eigvalsh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_matrix_power_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linspace_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logdet_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_matrix_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_max_reduction_no_dim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_minimum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nanquantile_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_new_full_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_conv1d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_max_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pad_circular_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nonzero_static_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_norm_fro_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ones_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_polygamma_polygamma_n_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_put_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_quantile_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_repeat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_reshape_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scatter_reduce_amin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_short_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_signal_windows_gaussian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_signal_windows_nuttall_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_bessel_y1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_spherical_bessel_j0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_spherical_bessel_j0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_split_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_squeeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sum_to_size_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tensordot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_vstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_cat_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_jiterator_4inputs_with_extra_args_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_linalg_vander_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_native_layer_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_ravel_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_signal_windows_blackman_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_squeeze_multiple_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_tensor_split_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_triu_indices_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out__refs_abs_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_atleast_3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_expand_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_eye_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_fftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_ihfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_lt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_minimum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_prelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_smooth_l1_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_multigammaln_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_squeeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_atan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_bitwise_xor_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out_cartesian_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cdouble_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cholesky_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_clamp_max_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cummin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_diagonal_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_empty_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_eye_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_fft_ifft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_flipud_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_jiterator_2inputs_2outputs_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_normalize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_maximum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_neg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_celu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_norm_inf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_outer_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_polygamma_polygamma_n_0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_qr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_addmv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_column_stack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_copysign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_hfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_inner_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_cholesky_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_eigh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_lu_factor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linspace_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_maximum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_norm_nuc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_permute_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_special_entr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_special_ndtri_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_split_with_sizes_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_tensordot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_triu_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_rsqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_laguerre_polynomial_l_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_unique_consecutive_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_int_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_atleast_2d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_bitwise_right_shift_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_conj_physical_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_dstack_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_erf_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_eye_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_fftn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_rfft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_geometric_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_isposinf_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_meshgrid_list_of_tensors_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_new_ones_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nextafter_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_dropout_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_glu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_hardshrink_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_signbit_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_square_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_std_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_tan_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_unbind_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_unbind_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_vsplit_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_acos_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_allclose_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_argsort_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_bitwise_right_shift_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_bucketize_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cholesky_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cholesky_inverse_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_dstack_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_expand_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_ifft2_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_igamma_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_index_select_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_int_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_isposinf_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_cond_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_eigh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_argmin_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_matrix_exp_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_max_reduction_no_dim_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_alpha_dropout_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_feature_alpha_dropout_with_train_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_interpolate_nearest-exact_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_max_unpool1d_grad_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_pairwise_distance_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nonzero_static_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_positive_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_repeat_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_round_decimals_neg_3_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_scatter_reduce_mean_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_sgn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_softmax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_polygamma_special_polygamma_n_0_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_tensordot_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_transpose_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_unique_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_unsqueeze_copy_cuda, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_acos_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_asin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atanh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_cos_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_digamma_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_div_no_rounding_mode_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_float_power_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_ldexp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_ldexp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_mvlgamma_mvlgamma_p_1_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_mvlgamma_mvlgamma_p_3_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_mvlgamma_mvlgamma_p_5_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_3_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_rsqrt_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sinh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_w_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_w_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_legendre_polynomial_p_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_zeta_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_tanh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_xlogy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_T_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cdouble_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_float_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_long_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_short_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acos_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acos_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcdiv_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_all_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amax_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_any_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_arange_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_arange_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asin_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asinh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bucketize_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_max_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_max_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clone_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_physical_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_physical_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumprod_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_embed_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_scatter_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dot_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_like_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_equal_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expm1_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eye_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eye_cuda_float8_e5m2, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftshift_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftshift_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftshift_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfftn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flatten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flatten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flip_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmod_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ge_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_heaviside_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_i0_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isinf_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isnan_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isposinf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_item_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lerp_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lgamma_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_cross_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_vector_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log1p_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_softmax_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_not_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logsumexp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lt_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_list_of_tensors_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mul_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nan_to_num_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_strided_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_ones_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_zeros_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_celu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_elu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_log_softmax_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_mish_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pairwise_distance_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pairwise_distance_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_shuffle_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_unshuffle_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmin_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmin_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmin_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softplus_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_positive_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_randn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_real_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reciprocal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_remainder_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rot90_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_round_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsqrt_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsub_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsub_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sigmoid_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_signbit_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinc_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_bessel_j1_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_bessel_j1_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_entr_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_erfcx_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i1_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_ndtri_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_ndtri_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_softmax_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_spherical_bessel_j0_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_square_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_multiple_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_take_along_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tensor_split_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trace_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_as_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_where_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_bitwise_xor_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_dot_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_fft_hfftn_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_floor_divide_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_index_select_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_logaddexp_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_logical_and_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_lt_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_nn_functional_huber_loss_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_nn_functional_margin_ranking_loss_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_t_copy_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_byte_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_chalf_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_chalf_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_chalf_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_float_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_half_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_half_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_half_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_abs_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_abs_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_add_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcmul_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addr_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_alias_copy_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_partial_views_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asinh_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_3d_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_or_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_tensors_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_tensors_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cauchy_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_chunk_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_chunk_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_max_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_physical_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_contiguous_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_contiguous_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cosh_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cumsum_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_deg2rad_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_embed_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_copy_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_copy_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_scatter_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_scatter_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_trunc_rounding_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dsplit_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dsplit_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dstack_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_like_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eq_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erf_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp2_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_copy_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expm1_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftn_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfftn_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft2_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft2_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft2_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fill_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fill_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flatten_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flatten_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flipud_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flipud_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_divide_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_divide_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmod_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmod_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_heaviside_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_i0_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_i0_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_add_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_copy_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_fill_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_select_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isinf_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isnan_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isreal_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isreal_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_item_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lcm_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_cross_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_vecdot_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_tensor_overload_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_tensor_overload_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log1p_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log2_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_and_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_or_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_xor_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_xor_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logsumexp_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lt_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lt_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lt_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_maximum_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_maximum_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mean_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_minimum_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_minimum_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_minimum_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nan_to_num_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_native_layer_norm_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_neg_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_zeros_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_elu_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_group_norm_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_layer_norm_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_margin_ranking_loss_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pairwise_distance_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_unshuffle_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_relu6_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmax_with_dtype_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmin_with_dtype_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softplus_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_normal__in_place_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_normal__in_place_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ones_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ones_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rad2deg_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rad2deg_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_randn_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ravel_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_renorm_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_repeat_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rot90_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rot90_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsqrt_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_select_scatter_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sigmoid_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sign_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_signbit_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sin_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinc_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinh_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_softmax_with_dtype_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_entr_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_entr_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_entr_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_3_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_3_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_3_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtr_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_ndtri_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_xlog1py_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_zeta_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_square_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_square_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_take_along_dim_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tan_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tanh_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_to_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_to_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trace_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tril_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_true_divide_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vdot_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_copy_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vsplit_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_where_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cdouble_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cfloat_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_double_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_double_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acosh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcmul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_alias_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_all_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_amin_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_any_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_arange_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asinh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_2d_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_not_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_not_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_tensors_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cat_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ceil_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clamp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_deg2rad_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_embed_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_no_rounding_mode_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_no_rounding_mode_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_trunc_rounding_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eq_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfc_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp2_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_as_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_hfftn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftshift_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fill_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fill_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flatten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flip_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flipud_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_divide_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_divide_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_divide_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_frac_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_frexp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gcd_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ge_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gt_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_heaviside_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_add_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_select_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isclose_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isposinf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_item_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lcm_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_le_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lerp_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lgamma_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_svdvals_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log1p_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_tensor_overload_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_maximum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_maximum_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_list_of_tensors_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_variadic_tensors_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_movedim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nan_to_num_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_strided_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_ones_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nextafter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_channel_shuffle_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_elu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hinge_embedding_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hinge_embedding_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_log_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_log_softmax_with_dtype_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pairwise_distance_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_shuffle_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ones_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ones_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ones_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_positive_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rad2deg_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ravel_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_as_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_roll_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_select_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sigmoid_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinc_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_bessel_j1_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_bessel_j1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_erfcx_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtr_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sqrt_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_to_size_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_t_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tanh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tensor_split_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tensor_split_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_to_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_to_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trace_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_triu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unflatten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_var_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_var_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vsplit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_where_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_where_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_zeros_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cdouble_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cdouble_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cdouble_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_chalf_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_chalf_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_char_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_float_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_long_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_long_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_long_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_short_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_abs_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acosh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_add_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addr_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addr_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_all_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_scatter_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_scatter_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atanh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_1d_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_not_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_or_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_right_shift_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_right_shift_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_xor_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_shapes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_to_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bucketize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ceil_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_column_stack_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_copysign_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_copysign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cos_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cosh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumprod_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_deg2rad_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_scatter_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_trunc_rounding_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expm1_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expm1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exponential_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfftn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfftn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fill_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flatten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flatten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fliplr_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fliplr_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_float_power_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_float_power_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_float_power_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmax_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmax_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_frac_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_frac_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ge_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_geometric_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_heaviside_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_heaviside_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hsplit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_i0_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_i0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_i0_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_igamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_add_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_add_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_select_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isinf_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isnan_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isreal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_svdvals_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_tensor_overload_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log10_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log1p_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_and_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_not_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logspace_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_masked_fill_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_mean_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_list_of_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_variadic_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_meshgrid_variadic_tensors_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ne_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_full_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_zeros_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nextafter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hardtanh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_huber_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_nll_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_unshuffle_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmax_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmin_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_triplet_margin_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_normal__in_place_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_normal_number_mean_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_real_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_real_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_repeat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_as_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rot90_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsub_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_select_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_signbit_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_signbit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_softmax_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_bessel_j0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_bessel_j1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1e_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_xlog1py_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_xlog1py_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_zeta_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_stack_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sub_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sub_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tan_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_triu_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_var_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_acos_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_addcdiv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_addcmul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_addr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_alias_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_all_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_atan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_char_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_double_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_dstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_expand_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_fft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_fftshift_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_hfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_ihfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_ihfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_rfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fliplr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_hsplit_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_jiterator_binary_return_by_ref_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_cholesky_ex_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_lu_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_lu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_vander_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_log1p_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logical_not_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_lt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_lu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_logsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_std_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_mm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_movedim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nan_to_num_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_new_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pad_replicate_negative_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pairwise_distance_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_rms_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nonzero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_permute_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_put_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_repeat_interleave_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_resize_as__cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_round_decimals_0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_signbit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_squeeze_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_std_unbiased_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_triangular_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_var_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_vstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_where_cuda_complex64, test/test_ops.py::TestCompositeComplianceCUDA::test_backward___radd___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward___rmul___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_acos_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cartesian_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_ifft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_ifftshift_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_eigh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_qr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_solve_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_log2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_median_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_normalize_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_movedim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_native_dropout_backward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_interpolate_nearest-exact_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_max_unpool3d_grad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_pad_replicate_negative_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_silu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_polar_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_special_i0e_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_special_ndtr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_split_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_squeeze_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_sub_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_take_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input___rsub___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_argmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_combinations_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_cosh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_dist_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_eq_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_expand_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_rfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_isposinf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_solve_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_svd_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_logical_xor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_masked_log_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_masked_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_cosine_embedding_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_dropout_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_instance_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_pinverse_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_round_decimals_0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_signal_windows_general_cosine_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_bessel_y1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_modified_bessel_i0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_squeeze_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_tile_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_transpose_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_uniform_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_view_as_complex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_atanh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_deg2rad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_rfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_rfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_float_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_gt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_isin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_ldl_factor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_logaddexp2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nanmean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_cross_entropy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_glu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_relu6_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_polygamma_polygamma_n_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_scatter_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_signal_windows_general_hamming_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_hermite_polynomial_h_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_i1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_take_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_unbind_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_addmm_decomposed_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_alias_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_angle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_as_strided_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_chunk_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_ifftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_gather_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_half_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_index_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_solve_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_logspace_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_logsumexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_msort_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_conv3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_embedding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_max_pool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_mish_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_repeat_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_rot90_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_sparse_mm_reduce_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_tensordot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_var_mean_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_abs_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_aminmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_as_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_bucketize_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_cholesky_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_clamp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_flip_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_gather_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_index_reduce_amin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_isclose_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_isin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_logaddexp2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_mH_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_var_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_adaptive_max_pool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_randn_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_ravel_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_real_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_resize__cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_resolve_neg_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_sigmoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_sin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_sinc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_bessel_j0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_tril_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_unbind_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_unfold_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_zero__cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_zeros_like_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_double_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_long_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_allclose_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_fftn_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_isreal_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_linalg_matrix_norm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_linalg_svdvals_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_log1p_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_new_empty_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_ravel_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_sin_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_unfold_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_asin_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_atanh_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_bool_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cholesky_solve_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cumsum_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_div_no_rounding_mode_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_einsum_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_expm1_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_fft_fftshift_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_fft_irfftn_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_flatten_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_isnan_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_jiterator_unary_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_cholesky_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_tensorsolve_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_logical_xor_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_long_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_masked_normalize_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_mul_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_mv_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nonzero_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_ones_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_randn_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_sqrt_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_squeeze_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_tan_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_unfold_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_unsafe_chunk_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_zeros_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_H_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view___radd___cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_acos_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_addr_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_clone_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_diag_embed_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_flip_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_logspace_tensor_overload_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_ne_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_permute_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_pow_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_real_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_sgn_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_std_mean_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_abs_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_as_strided_partial_views_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_bmm_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_cross_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_diagonal_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_half_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_imag_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_jiterator_binary_return_by_ref_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_cholesky_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_cond_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_det_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_solve_ex_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_log1p_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_masked_mean_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_pad_replicate_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_pairwise_distance_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_triplet_margin_loss_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_permute_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_ravel_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_reciprocal_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_repeat_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_t_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_trace_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_unbind_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_zeros_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view__native_batch_norm_legit_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_arange_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_as_strided_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_clamp_min_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_div_trunc_rounding_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_exponential_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fft_ihfftn_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_flipud_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_linalg_cross_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_log_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_new_empty_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_mish_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_triplet_margin_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_pow_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_repeat_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_reshape_as_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_sinc_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_special_i0e_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_special_softmax_with_dtype_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_tensor_split_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_tril_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_vstack_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_div_floor_rounding_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_dstack_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_exp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_expand_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fft_fftshift_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_float_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_hypot_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_jiterator_binary_return_by_ref_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_log_normal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_logsumexp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_max_reduction_with_dim_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_minimum_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_mvlgamma_mvlgamma_p_1_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_alpha_dropout_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_binary_cross_entropy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_cross_entropy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_embedding_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_instance_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_max_pool2d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nonzero_static_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_normal_in_place_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_polar_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_rad2deg_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_sgn_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_short_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_erfcx_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_hermite_polynomial_he_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_i1e_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_shifted_chebyshev_polynomial_v_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_std_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_sum_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_take_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_unique_consecutive_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_unsafe_split_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake___rand___cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake___rmatmul___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake__unsafe_masked_index_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast__segment_reduce_offsets_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_atan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cfloat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_copysign_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_empty_strided_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_erfinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_fftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_float_power_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_gt_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_half_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_inner_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_logical_not_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_lt_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_cumsum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_log_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_conv3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_real_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_repeat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_scatter_reduce_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_select_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signal_windows_blackman_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signal_windows_kaiser_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_erfcx_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_hermite_polynomial_h_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_unsqueeze_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_bitwise_left_shift_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_broadcast_to_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_chunk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_clamp_max_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_cov_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_angle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_cdist_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_exp2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_fft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_cholesky_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_diagonal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_eigvalsh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_svdvals_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_log10_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_log_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_masked_sum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_matmul_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_median_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_cross_entropy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_gelu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_norm_fro_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_renorm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_round_decimals_neg_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_split_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_std_mean_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_vstack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp___rsub___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_bfloat16_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cartesian_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cdist_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_clamp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_div_trunc_rounding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_expm1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_fft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fmod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_lerp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_lgamma_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_cholesky_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_norm_subgradients_at_zero_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_native_dropout_backward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_cosine_embedding_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_interpolate_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_logsigmoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_smooth_l1_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_ormqr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_qr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_reshape_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_round_decimals_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_sparse_mm_reduce_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_special_i1e_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_svd_lowrank_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_tensor_split_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_trunc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_cummax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_empty_permuted_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_rfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fmod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_hsplit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_diagonal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_log2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logaddexp2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_lu_unpack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_movedim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nanmean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_avg_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_conv3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_max_unpool3d_grad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_normal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_normal_in_place_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_pinverse_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_scatter_add_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_ndtri_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_split_with_sizes_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_std_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_take_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_tril_indices_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_triu_indices_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_unique_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_unsafe_chunk_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_vsplit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_H_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops___radd___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_clamp_min_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_combinations_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_diagonal_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_eq_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_hfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_ihfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_irfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_qr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_matrix_exp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_minimum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_multinomial_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_new_zeros_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_linear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_max_unpool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nonzero_static_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_quantile_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_real_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_remainder_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_slice_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_zeta_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_split_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_std_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_take_along_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_arange_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_complex128, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_ones_cuda_complex128, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_ones_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_zeros_cuda_complex32, test/test_ops.py::TestTagsCUDA::test_tags__refs_bitwise_right_shift_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags__refs_constant_pad_nd_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_dsplit_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_fft_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_rfft_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_logical_or_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_neg_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_new_full_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_softmax_with_dtype_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_all_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_any_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_as_strided_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_cosh_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_cumprod_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_diag_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_diagonal_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_dist_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_erf_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_erfinv_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_gcd_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags_half_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_kthvalue_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_cholesky_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_eigvals_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_ldl_factor_ex_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_solve_ex_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linspace_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_logaddexp2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_gelu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_hardsigmoid_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_smooth_l1_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_scalar_tensor_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_signal_windows_nuttall_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_bessel_y0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_triangular_solve_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_unique_cuda_float32
2025-07-17T09:39:23.7026359Z 
2025-07-17T09:44:55.9989224Z 
2025-07-17T09:44:55.9990304Z test_ops 16/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_16.20_49ae16410a0eed75_.log
2025-07-17T09:44:56.0510039Z Running 1712 items in this shard: test/test_ops.py::TestCommonCUDA::test_compare_cpu_H_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu___rmatmul___cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_alias_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_fmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_vstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_atleast_1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_bool_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_index_reduce_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_ldexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_masked_normalize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_multinomial_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_native_batch_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_max_unpool3d_grad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_xlogy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_as_strided_partial_views_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_clone_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_isfinite_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_new_ones_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_permute_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_real_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_resolve_neg_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_tan_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_where_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes__chunk_cat_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_bitwise_xor_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_erf_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_exponential_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_ihfft2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_rfftn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_geometric_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_log_softmax_with_dtype_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_poisson_nll_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_prod_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_real_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_true_divide_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_view_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__segment_reduce_offsets_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_abs_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_baddbmm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_broadcast_tensors_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_deg2rad_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_dstack_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_empty_strided_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_heaviside_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_index_fill_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_isposinf_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_eigh_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_ldl_factor_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_qr_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_log2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_log_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_select_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nanquantile_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_narrow_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_ne_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_new_empty_strided_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_adaptive_avg_pool2d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_group_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_layer_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_max_unpool1d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_multi_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_normalize_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_pixel_unshuffle_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_rms_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nonzero_static_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_normal_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_polygamma_polygamma_n_2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_renorm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_scatter_reduce_mean_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_select_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_softmax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_bessel_y0_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_ndtri_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_split_with_sizes_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_stack_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_svd_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_torch_ops_aten__safe_softmax_default_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_var_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_var_mean_unbiased_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_view_cuda, test/test_ops.py::TestCommonCUDA::test_errors___rand___cuda, test/test_ops.py::TestCommonCUDA::test_errors___rmod___cuda, test/test_ops.py::TestCommonCUDA::test_errors__chunk_cat_cuda, test/test_ops.py::TestCommonCUDA::test_errors_as_strided_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_errors_bitwise_and_cuda, test/test_ops.py::TestCommonCUDA::test_errors_cat_cuda, test/test_ops.py::TestCommonCUDA::test_errors_cauchy_cuda, test/test_ops.py::TestCommonCUDA::test_errors_div_floor_rounding_cuda, test/test_ops.py::TestCommonCUDA::test_errors_gt_cuda, test/test_ops.py::TestCommonCUDA::test_errors_logaddexp_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_soft_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_errors_normal_in_place_cuda, test/test_ops.py::TestCommonCUDA::test_errors_signal_windows_exponential_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_sum_layout2_cuda, test/test_ops.py::TestCommonCUDA::test_errors_special_shifted_chebyshev_polynomial_u_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_asin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_cummax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_cummin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_fft_irfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_frac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_igamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_index_reduce_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_index_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_logcumsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_logical_xor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_norm_inf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_ormqr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_entr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_i1e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices___rand___cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_argwhere_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_as_strided_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_bitwise_left_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_column_stack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_copysign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_diff_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_digamma_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_dist_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_dstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_fft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_ifft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fill_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_index_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_index_put_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_index_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_isfinite_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_kron_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_cross_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_householder_product_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_inv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_log10_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logaddexp2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_argmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_std_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mul_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_narrow_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_hardsigmoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_one_hot_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_norm_nuc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_resolve_conj_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_scatter_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_select_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_hermite_polynomial_h_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_i0e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_log_ndtr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_modified_bessel_i1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_split_list_args_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_squeeze_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tile_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_topk_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_trace_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_triu_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_vdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_vstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_H_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_alias_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_as_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_asinh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_cat_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_conj_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_dstack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_empty_permuted_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_exp2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fft_rfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_fmax_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_le_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_new_empty_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_nn_functional_pad_circular_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_polygamma_polygamma_n_3_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_sort_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_squeeze_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_tile_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_to_sparse_cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___rpow___cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_acos_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addcmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_arange_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atan2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_baddbmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bitwise_xor_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bool_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_conj_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_copysign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cummin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diagflat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diff_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_digamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_erf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_fft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ifft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_irfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_rfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fmin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_geqrf_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_isfinite_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_isposinf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_isreal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_item_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_jiterator_4inputs_with_extra_args_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_le_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_cross_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_slogdet_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_tensorinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_vecdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logical_and_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_lu_unpack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_softmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_min_binary_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_elu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_glu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_l1_loss_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pad_reflect_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pad_replicate_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pdist_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_threshold_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_norm_nuc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_pca_lowrank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_polygamma_polygamma_n_2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_qr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_reciprocal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_renorm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_resize__cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_rot90_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_rsqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sin_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sinh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_bessel_j1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_i1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_trapz_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_trapz_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unique_consecutive_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unsafe_chunk_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unsafe_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unsqueeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_view_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_zero__cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_broadcast_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_linalg_tensorsolve_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_linalg_tensorsolve_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_meshgrid_variadic_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_one_hot_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_squeeze_multiple_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_unbind_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_unbind_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out___rmatmul___cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_addr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_alias_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_diagonal_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_dsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_fft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_le_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_positive_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_sgn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_tril_indices_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out__refs_true_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_all_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_aminmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_bucketize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_conj_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cov_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cumulative_trapezoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_floor_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_ge_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_index_reduce_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_ldexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_lerp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_tensorsolve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_log1p_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_matrix_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_new_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_max_unpool2d_grad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_selu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_norm_fro_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_ormqr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_rad2deg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_baddbmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_baddbmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_deg2rad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_index_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_lerp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_eigh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_multi_dot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_svd_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_tensorsolve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_vector_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nn_functional_normalize_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_polygamma_polygamma_n_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_reciprocal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_renorm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_special_i0e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_take_along_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_resolve_conj_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_scatter_reduce_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_sgn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_signal_windows_nuttall_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_ndtri_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_to_sparse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_triu_indices_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out_unbind_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_unsafe_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_vsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning___radd___cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_empty_strided_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_ifftshift_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fmax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_gcd_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_masked_fill_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_native_layer_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_prelu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_selu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_sum_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_view_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_arange_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_as_strided_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_block_diag_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_conj_physical_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_copysign_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cumprod_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_erfinv_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_eye_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_frac_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_frexp_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_gcd_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_gradient_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_heaviside_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_multi_dot_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linspace_tensor_overload_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_log_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_mean_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_adaptive_avg_pool2d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_adaptive_max_pool3d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_avg_pool1d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_cosine_embedding_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_hardsigmoid_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_interpolate_bicubic_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_interpolate_trilinear_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_max_pool2d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_ormqr_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_polar_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_ravel_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_scatter_reduce_prod_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_signal_windows_hann_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_scaled_modified_bessel_k0_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_shifted_chebyshev_polynomial_t_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_split_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_t_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_trapz_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_unbind_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_vstack_cuda, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_copysign_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_copysign_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_cos_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_exp_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_expm1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_lgamma_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log10_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log1p_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_logit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_mvlgamma_mvlgamma_p_3_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_0_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_1_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sigmoid_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sinc_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_t_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_v_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_v_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_laguerre_polynomial_l_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_w_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sqrt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bfloat16_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cdouble_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_double_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_float_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_float_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_half_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_int_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_long_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_short_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_add_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcdiv_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcmul_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_alias_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_partial_views_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asin_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_asinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_2d_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_3d_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_and_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_or_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_xor_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_block_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_tensors_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_to_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bucketize_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cat_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_chunk_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clamp_min_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_constant_pad_nd_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_contiguous_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cos_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_deg2rad_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_deg2rad_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_scatter_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_digamma_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_trunc_rounding_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_like_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_equal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erf_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erfinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftshift_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftshift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfftn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flatten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flip_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fliplr_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flipud_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flipud_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flipud_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_float_power_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_float_power_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_float_power_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_frac_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hsplit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_fill_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isclose_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isfinite_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isnan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isneginf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isposinf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lerp_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lgamma_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_matrix_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_svd_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_svdvals_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_tensor_overload_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log_normal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_not_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_or_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_tensor_overload_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logsumexp_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logsumexp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lt_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_masked_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_variadic_tensors_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_variadic_tensors_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_variadic_tensors_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_minimum_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nan_to_num_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_celu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_channel_shuffle_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_channel_shuffle_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_group_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_huber_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_margin_ranking_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_mse_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pairwise_distance_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pairwise_distance_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_shuffle_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_unshuffle_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_poisson_nll_loss_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_relu6_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_relu_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmin_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmin_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ones_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_positive_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_real_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_real_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_renorm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_renorm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_repeat_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_roll_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rot90_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_signbit_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_erfcx_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_ndtr_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_logit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_1_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_square_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_std_mean_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trace_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trace_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trunc_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_trunc_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_xlogy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_exponential_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_eye_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_fft_ihfft2_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_fft_ihfft_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_nn_functional_triplet_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_pow_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_renorm_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_vdot_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bool_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_byte_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cdouble_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cdouble_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cfloat_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_double_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_half_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_int_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addr_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_alias_copy_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_alias_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_all_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_all_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_scatter_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asinh_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asinh_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asinh_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asinh_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_left_shift_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_xor_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_block_diag_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_block_diag_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_tensors_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_to_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cat_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cat_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ceil_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_column_stack_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_physical_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_constant_pad_nd_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cumprod_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cumsum_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_scatter_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_scatter_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dstack_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eq_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_equal_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_erfinv_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp2_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_as_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftshift_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft2_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfft_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_hfftn_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifftn_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft2_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfftn_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_rfft_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fill_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flatten_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flip_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_float_power_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_divide_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_frexp_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_geometric_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_gt_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hsplit_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hstack_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_igammac_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_copy_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_fill_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_fill_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_select_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isclose_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isfinite_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isfinite_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isinf_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isnan_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isposinf_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lcm_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_le_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_le_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_diagonal_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_matrix_norm_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_vector_norm_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_tensor_overload_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_tensor_overload_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log1p_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log2_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_normal_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_softmax_with_dtype_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lt_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lt_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_maximum_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_maximum_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_maximum_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_movedim_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_narrow_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_neg_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_neg_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_full_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_ones_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_ones_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_ones_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_alpha_dropout_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_alpha_dropout_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hardtanh_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hardtanh_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_huber_loss_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_shuffle_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_shuffle_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_unshuffle_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_prelu_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_relu6_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_relu6_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmax_with_dtype_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_triplet_margin_loss_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_triplet_margin_loss_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ones_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rad2deg_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ravel_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_real_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reciprocal_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rot90_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsub_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sigmoid_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sigmoid_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sigmoid_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sign_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_softmax_with_dtype_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_bessel_j0_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_bessel_j0_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_bessel_j1_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i0e_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i0e_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_ndtr_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_softmax_with_dtype_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_softmax_with_dtype_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_logit_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_1_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_split_with_sizes_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_to_size_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_to_size_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_take_along_dim_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_take_along_dim_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trace_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trace_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tril_indices_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_true_divide_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_copy_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vsplit_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vstack_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_where_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_xlogy_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_zeros_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_cfloat_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_half_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_int_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_long_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_abs_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acos_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acosh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addcmul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_all_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_allclose_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_allclose_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_any_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_scatter_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_right_shift_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_xor_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_block_diag_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cat_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cauchy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_physical_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cosh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_count_nonzero_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumprod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumprod_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_deg2rad_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_no_rounding_mode_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_trunc_rounding_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dot_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dot_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dot_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eq_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eq_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_equal_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_equal_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_equal_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erf_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfc_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exponential_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfftn_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flip_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flip_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fliplr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_frexp_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gcd_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_gt_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hstack_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hypot_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_igamma_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isfinite_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isneginf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isneginf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isposinf_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isposinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lgamma_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_vector_norm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linspace_tensor_overload_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log10_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log1p_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log1p_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log1p_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logaddexp2_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_variadic_tensors_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_minimum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mul_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_strided_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_full_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_alpha_dropout_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_channel_shuffle_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hardshrink_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_margin_ranking_loss_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_threshold_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ones_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reciprocal_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reciprocal_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_remainder_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_renorm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_repeat_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_repeat_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rot90_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsub_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsub_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sgn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_signbit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_signbit_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_bessel_j0_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_erfcx_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_logit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_3_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtr_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sqrt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sqrt_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_multiple_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_std_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_to_size_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tensor_split_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_true_divide_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_true_divide_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trunc_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unflatten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_zeros_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_T_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_byte_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_byte_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cfloat_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cfloat_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_chalf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_chalf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_char_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_int_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_short_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addcmul_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_all_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_all_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amax_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_scatter_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_scatter_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_1d_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_1d_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_2d_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_and_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_left_shift_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_not_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_xor_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_block_diag_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ceil_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_max_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_min_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_column_stack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_physical_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_copysign_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_count_nonzero_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_count_nonzero_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumprod_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_embed_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_scatter_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_no_rounding_mode_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_no_rounding_mode_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_strided_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_strided_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_strided_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eq_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_equal_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_as_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expm1_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expm1_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exponential_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfftn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfftn_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fill_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fliplr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flipud_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmax_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmin_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_frexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ge_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hsplit_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hsplit_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_igammac_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_fill_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isinf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isinf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isinf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isneginf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isposinf_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isposinf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_item_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_item_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lerp_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lerp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_diagonal_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_svdvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log10_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log1p_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log1p_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_softmax_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logaddexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_and_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_not_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_xor_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_xor_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_xor_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logspace_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logspace_tensor_overload_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logspace_tensor_overload_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logsumexp_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lt_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_maximum_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_minimum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_minimum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_mul_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hinge_embedding_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_log_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_nll_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pdist_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_prelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_threshold_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_normal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rad2deg_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_randn_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ravel_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_real_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_repeat_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rot90_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_select_scatter_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sgn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sgn_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sgn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sin_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinc_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_softmax_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_erfcx_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_logit_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_3_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_5_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtr_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtri_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_softmax_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_spherical_bessel_j0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_split_with_sizes_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_split_with_sizes_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_multiple_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_multiple_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_std_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sub_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_to_size_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_to_size_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tan_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tanh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tanh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_to_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_to_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_triu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_true_divide_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_true_divide_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_var_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_xlogy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_zeros_cuda_float16, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_H_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_T_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___getitem___cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___rmatmul___cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager__chunk_cat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager__unsafe_masked_index_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager__unsafe_masked_index_put_accumulate_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_atleast_3d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_broadcast_tensors_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cfloat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cholesky_inverse_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_div_no_rounding_mode_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_exp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_expand_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_ifftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_flipud_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_floor_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_gradient_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_hstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_index_select_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_det_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_amax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_meshgrid_variadic_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_ne_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_adaptive_max_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_interpolate_nearest-exact_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_max_unpool1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pad_constant_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pixel_shuffle_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_normal_in_place_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_pca_lowrank_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_polar_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_positive_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_real_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_resize_as__cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_round_decimals_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_scatter_reduce_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sparse_sampled_addmm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_split_with_sizes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_take_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unsafe_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_view_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward__batch_norm_with_update_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_asinh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_complex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_constant_pad_nd_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cross_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_div_floor_rounding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_dsplit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_ihfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_grid_sampler_2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_cholesky_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_svdvals_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_logdet_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_amin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_log_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_masked_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_embedding_bag_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_embedding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_gaussian_nll_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_rrelu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_unfold_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_permute_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_rot90_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_rsqrt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_vstack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_where_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_aminmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_any_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_as_strided_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_baddbmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_count_nonzero_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_exp2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_expand_as_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_fft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_irfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_gt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_ldexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_ldl_factor_ex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_log10_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_logical_or_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_lt_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_masked_std_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_min_binary_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_elu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_kl_div_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_soft_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_normal_in_place_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_polygamma_polygamma_n_3_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_short_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_sigmoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_signal_windows_gaussian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_signal_windows_general_hamming_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_unique_consecutive_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad___radd___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_cdouble_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_chalf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_char_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_erfinv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_exponential_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_ihfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_rfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_igamma_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_isreal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_ldexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_slogdet_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_softmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_matrix_exp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nansum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_native_batch_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_new_full_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_interpolate_nearest_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_max_pool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_ones_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_rand_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sgn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_signal_windows_cosine_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_std_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_unflatten_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_unsqueeze_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_addbmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_addr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_bernoulli_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_ceil_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_chalf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_clone_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_div_floor_rounding_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_exponential_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_eye_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_flipud_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_frexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_ge_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_igammac_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_logaddexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_masked_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_minimum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_gelu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_hardswish_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_linear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_logsigmoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_soft_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_norm_nuc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_quantile_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_randint_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_resolve_conj_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_signal_windows_blackman_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_entr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_var_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay___rmul___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_angle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_bernoulli_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_cdouble_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_contiguous_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_cos_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_cummin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_cumsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_double_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_ifftshift_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_irfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fill_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_floor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_index_reduce_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_index_select_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_jiterator_2inputs_2outputs_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_jiterator_unary_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_logical_not_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_logsumexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_normalize_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nanmedian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_gelu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_round_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_signal_windows_bartlett_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_signal_windows_cosine_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_signal_windows_hamming_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_shifted_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_std_mean_unbiased_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_transpose_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_trapz_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_vstack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_zeros_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view___getitem___cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_any_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_as_strided_partial_views_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_empty_like_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_exp2_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_exp_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_hfftn_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fliplr_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_isclose_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_nn_functional_pixel_unshuffle_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_rsub_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_special_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_sum_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_triu_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_unflatten_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__unsafe_masked_index_put_accumulate_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_as_strided_partial_views_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_block_diag_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_broadcast_to_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cholesky_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cholesky_inverse_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_diagonal_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_diagonal_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_dist_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_fft_hfft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_fft_ifftshift_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_fft_irfft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_flipud_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_geqrf_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_gradient_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_hstack_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_isclose_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_isfinite_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_lu_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_vector_norm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_masked_mean_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_masked_sum_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_matrix_exp_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_conv_transpose2d_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_feature_alpha_dropout_without_train_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_l1_loss_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_pad_replicate_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nonzero_static_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_norm_nuc_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_normal_in_place_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_qr_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_scatter_add_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_select_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_stft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_addcmul_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_allclose_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_as_strided_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_atanh_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_div_no_rounding_mode_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_empty_like_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fft_irfft_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fft_irfftn_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_istft_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_masked_fill_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_meshgrid_list_of_tensors_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_neg_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_nn_functional_log_softmax_with_dtype_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_t_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_atleast_2d_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_bool_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_combinations_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_eye_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fft_irfft2_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_geqrf_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_isfinite_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_solve_triangular_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_log2_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_logical_not_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_mul_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_mv_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_narrow_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_new_ones_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_outer_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_qr_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_rsqrt_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_std_mean_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_take_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_tensordot_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_unsafe_chunk_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_var_mean_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs__conversions_char_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs__conversions_polar_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_clamp_max_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_column_stack_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_equal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_erf_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_erfc_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fft_hfft2_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fmax_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fmin_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_logical_and_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_celu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_margin_ranking_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_special_multigammaln_mvlgamma_p_1_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_square_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_transpose_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__upsample_bilinear2d_aa_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_clamp_min_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_diff_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_erfinv_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_full_like_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_int_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_lu_factor_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_linalg_pinv_singular_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_log10_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_celu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_max_unpool2d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_nll_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_norm_fro_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_polygamma_polygamma_n_1_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_polygamma_polygamma_n_4_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_repeat_interleave_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_rot90_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_searchsorted_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_entr_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_log_ndtr_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_t_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_to_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_transpose_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_vdot_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_view_as_complex_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake_arange_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast__segment_reduce_lengths_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_aminmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_angle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_atleast_2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cummax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_diagonal_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_geometric_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_igamma_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_index_put_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_multi_dot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_norm_subgradients_at_zero_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_pinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_masked_logsumexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_native_dropout_backward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_new_zeros_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_prelu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_triplet_margin_with_distance_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_ones_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_resize__cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_scalar_tensor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signbit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_sin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_split_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_sub_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_trace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_true_divide_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_trunc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_unfold_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_char_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp___rsub___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_acos_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_addcmul_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_as_strided_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_cov_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_diagonal_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_digamma_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_dot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_dsplit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_ihfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_gather_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_index_reduce_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_index_reduce_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_native_layer_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_embedding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_instance_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_scaled_dot_product_attention_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_select_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_sigmoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_special_i1e_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_squeeze_multiple_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_triangular_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_view_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_addbmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_as_strided_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_cholesky_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_constant_pad_nd_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_diag_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_ihfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_min_binary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_max_pool1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_split_list_args_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_square_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_take_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_to_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_unsqueeze_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_hfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_float_power_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_full_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_ldl_factor_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_lu_factor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_solve_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_vector_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logcumsumexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_logical_and_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_median_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_max_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_threshold_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_reciprocal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_resolve_conj_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_rsub_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_scatter_reduce_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_sgn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_log_ndtr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops___rdiv___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops___rsub___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_alias_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_atleast_3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_cartesian_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_cauchy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_char_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_conj_physical_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_div_trunc_rounding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_einsum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_expand_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_rfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_flipud_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_geometric_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_hypot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_isneginf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linspace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_log_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nan_to_num_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_narrow_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_celu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_conv_transpose1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_dropout3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_interpolate_nearest_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_pixel_unshuffle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_sign_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_signal_windows_cosine_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_signal_windows_general_hamming_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_signal_windows_hann_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_entr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_ndtr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_ndtri_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_scaled_modified_bessel_k1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_stack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_std_mean_unbiased_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_stft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_to_sparse_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_unsqueeze_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_tensor_overload_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_cuda_bfloat16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_tensor_overload_cuda_bfloat16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_ones_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_arange_cuda_float16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_tensor_overload_cuda_bfloat16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_tensor_overload_cuda_float64, test/test_ops.py::TestTagsCUDA::test_tags___rdiv___cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_broadcast_shapes_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_clamp_min_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_diagonal_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_erfc_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_ifft_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_floor_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_floor_divide_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_hstack_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_linspace_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_elu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_softplus_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_norm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_rad2deg_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_tan_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_view_as_complex_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_view_as_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_bmm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_broadcast_to_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_deg2rad_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_exponential_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_isclose_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_logical_and_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_logical_xor_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_amin_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_softmin_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_max_reduction_with_dim_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_neg_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_new_zeros_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_interpolate_nearest-exact_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_mish_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_pairwise_distance_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_softsign_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_permute_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_polygamma_polygamma_n_2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_reshape_as_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_reshape_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_roll_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_signal_windows_blackman_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_modified_bessel_k0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_squeeze_multiple_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_to_sparse_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_vdot_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_zeros_like_cuda_float32
2025-07-17T09:44:56.0971957Z 
2025-07-17T09:44:56.0972408Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T09:44:56.0972837Z Uploading artifacts took 0.00 seconds
2025-07-17T09:46:44.9139393Z 
2025-07-17T09:46:44.9140499Z test_ops_gradients 3/3 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_gradients_3.3_83618f1301a98274_.log
2025-07-17T09:46:44.9714907Z Running 1824 items in this shard: test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_H_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_NumpyCubeCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_NumpyNonzeroCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_NumpySplitCopyCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_T_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_T_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad___getitem___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad___getitem___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad___radd___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad___rdiv___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad___rpow___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad___rsub___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad__unsafe_masked_index_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad__unsafe_masked_index_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad__unsafe_masked_index_put_accumulate_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad__unsafe_masked_index_put_accumulate_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad__upsample_bilinear2d_aa_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_add_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_add_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_addmm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_addmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_addmm_decomposed_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_addmv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_all_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_angle_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_argmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_argmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_argwhere_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_as_strided_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_as_strided_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_as_strided_partial_views_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_asin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_atan_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_atan_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_atanh_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_atleast_1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_atleast_2d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_atleast_3d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_bfloat16_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_bool_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_cfloat_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_cholesky_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_cholesky_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_clamp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_clamp_min_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_column_stack_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_combinations_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_combinations_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_conj_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_conj_physical_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_conj_physical_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_contiguous_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_contiguous_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_corrcoef_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_count_nonzero_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_cov_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_cross_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_cumprod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_cumsum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_deg2rad_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_diag_embed_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_diag_embed_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_diagonal_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_diagonal_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_diagonal_scatter_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_diff_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_div_no_rounding_mode_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_dot_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_double_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_double_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_dstack_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_einsum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_einsum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_empty_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_empty_like_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_empty_like_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_empty_permuted_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_equal_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_equal_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_exp2_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_exp2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_expand_as_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_expand_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_expand_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_expm1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_exponential_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_fft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_fftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_fftshift_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_fftshift_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_hfft2_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_hfft2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_hfftn_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_ifft2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_ifft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_ifftn_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_ifftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_ihfft2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_ihfftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_irfft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fft_rfftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fill_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_flatten_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fliplr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fliplr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_flipud_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_float_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_fmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_frexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_full_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_full_like_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_geometric_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_heaviside_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_hsplit_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_hypot_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_index_add_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_index_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_index_fill_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_index_reduce_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_int_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_isfinite_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_isinf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_isnan_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_isneginf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_isposinf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_isreal_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_istft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_item_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_jiterator_binary_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_kthvalue_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_lgamma_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_cholesky_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_cholesky_ex_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_cholesky_ex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_det_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_diagonal_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_eigvals_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_eigvalsh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_inv_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_inv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_ldl_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_lstsq_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_lu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_lu_factor_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_lu_factor_ex_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_lu_factor_ex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_lu_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_matrix_rank_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_multi_dot_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_pinv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_pinv_singular_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_qr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_qr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_slogdet_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_svd_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_tensorinv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_linalg_vander_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_log10_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_log10_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_log1p_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_log_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_log_softmax_with_dtype_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_logaddexp2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_logcumsumexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_logical_and_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_logical_not_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_logspace_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_logspace_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_logsumexp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_lu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_lu_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_mH_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_amax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_argmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_cumprod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_cumsum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_logaddexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_logsumexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_normalize_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_select_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_softmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_softmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_sum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_masked_var_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_matrix_exp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_max_binary_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_max_pool2d_with_indices_backward_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_meshgrid_variadic_tensors_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_min_binary_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_min_reduction_no_dim_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_minimum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_mvlgamma_mvlgamma_p_3_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_mvlgamma_mvlgamma_p_5_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nan_to_num_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nanmedian_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nansum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_narrow_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_native_dropout_backward_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_native_layer_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_ne_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_new_empty_strided_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_new_ones_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_new_zeros_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_adaptive_avg_pool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_adaptive_avg_pool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_adaptive_avg_pool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_adaptive_max_pool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_adaptive_max_pool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_avg_pool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_batch_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_binary_cross_entropy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_binary_cross_entropy_with_logits_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_channel_shuffle_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_conv1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_conv2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_conv3d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_conv3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_conv_transpose1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_conv_transpose2d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_dropout3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_gelu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_glu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_grid_sample_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_hardshrink_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_hardtanh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_instance_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_interpolate_bicubic_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_interpolate_bilinear_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_interpolate_nearest_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_interpolate_trilinear_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_kl_div_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_local_response_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_max_pool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_max_pool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_max_unpool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_mish_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_multi_margin_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_normalize_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_pad_constant_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_pairwise_distance_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_pixel_shuffle_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_pixel_unshuffle_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_poisson_nll_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_rms_norm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_smooth_l1_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_softplus_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_softsign_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_softsign_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_tanhshrink_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_triplet_margin_with_distance_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nn_functional_upsample_nearest_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_nonzero_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_norm_fro_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_norm_inf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_norm_nuc_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_normal_in_place_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_ones_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_ones_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_ones_like_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_outer_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_pca_lowrank_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_permute_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_pinverse_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_polygamma_polygamma_n_3_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_polygamma_polygamma_n_4_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_positive_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_prod_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_prod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_qr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_rad2deg_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_randint_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_randn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_randn_like_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_ravel_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_ravel_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_reciprocal_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_remainder_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_renorm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_repeat_interleave_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_reshape_as_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_reshape_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_reshape_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_resize__cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_resolve_conj_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_resolve_neg_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_resolve_neg_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_roll_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_roll_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_round_decimals_0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_rsqrt_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_scatter_add_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_scatter_reduce_amax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_scatter_reduce_amin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_scatter_reduce_prod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_searchsorted_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_select_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_sgn_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_short_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_sigmoid_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_signal_windows_bartlett_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_signal_windows_general_hamming_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_signal_windows_kaiser_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_sin_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_sinc_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_slice_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_softmax_with_dtype_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_sort_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_sparse_mm_reduce_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_bessel_j1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_bessel_y0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_chebyshev_polynomial_t_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_chebyshev_polynomial_u_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_entr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_hermite_polynomial_h_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_legendre_polynomial_p_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_log_ndtr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_modified_bessel_k0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_ndtr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_scaled_modified_bessel_k0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_shifted_chebyshev_polynomial_t_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_special_shifted_chebyshev_polynomial_v_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_split_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_split_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_split_list_args_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_split_with_sizes_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_sqrt_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_square_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_squeeze_multiple_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_stft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_sub_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_svd_lowrank_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_t_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_t_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_tile_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_tile_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_to_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_to_sparse_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_to_sparse_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_transpose_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_transpose_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_trapezoid_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_triangular_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_true_divide_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_unfold_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_unfold_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_uniform_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_unsafe_chunk_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_unsafe_chunk_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_unsqueeze_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_vdot_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_view_as_real_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_view_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_vstack_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_vstack_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_xlogy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_fail_gradgrad_zeros_like_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_NumpyCubeCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_NumpyMulScalarCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_NumpySortCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_NumpySplitCopyWithIntCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_NumpyTakeCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad___getitem___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad___rmul___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad___rpow___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad___rsub___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad__batch_norm_with_update_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad__chunk_cat_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad__softmax_backward_data_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad__unsafe_masked_index_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad__upsample_bilinear2d_aa_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_acos_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_acosh_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_add_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_addbmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_addcdiv_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_addcmul_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_addmm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_addmm_decomposed_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_all_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_amax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_amin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_angle_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_any_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_arange_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_argsort_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_argwhere_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_as_strided_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_as_strided_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_as_strided_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_as_strided_partial_views_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_asinh_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_atan_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_atanh_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_atleast_1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_baddbmm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_bmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_broadcast_to_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_bucketize_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_cfloat_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_char_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_cholesky_inverse_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_cholesky_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_chunk_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_clamp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_column_stack_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_column_stack_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_conj_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_constant_pad_nd_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_contiguous_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_contiguous_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_cosh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_count_nonzero_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_cross_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_cummax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_cumsum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_cumsum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_diag_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_diagflat_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_diagonal_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_diagonal_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_diagonal_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_diff_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_diff_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_dist_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_dsplit_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_dstack_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_empty_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_empty_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_empty_like_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_eq_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_erfinv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_exp2_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_eye_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_fft2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_fft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_fftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_fftshift_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_fftshift_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_hfft2_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_hfft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_hfftn_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_hfftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_ifft2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_ifft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_ifft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_ifftshift_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_ihfft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_irfft2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fft_irfft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_flatten_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_flip_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fliplr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_float_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_float_power_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_fmod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_full_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_full_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_full_like_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_gt_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_heaviside_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_hsplit_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_hsplit_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_igammac_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_index_add_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_index_add_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_index_fill_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_index_reduce_amin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_inner_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_int_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_invoke_quant_packed_simple_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_isclose_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_isfinite_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_isnan_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_isnan_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_isreal_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_istft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_item_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_jiterator_4inputs_with_extra_args_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_jiterator_binary_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_jiterator_binary_return_by_ref_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_jiterator_unary_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_kron_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_kthvalue_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_ldexp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_ldexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_le_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_lgamma_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_eigvals_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_eigvalsh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_inv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_ldl_factor_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_ldl_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_lstsq_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_lu_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_lu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_lu_factor_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_lu_factor_ex_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_lu_factor_ex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_matrix_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_matrix_power_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_matrix_rank_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_norm_subgradients_at_zero_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_pinv_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_pinv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_pinv_hermitian_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_pinv_singular_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_qr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_solve_ex_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_solve_ex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_tensorinv_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_tensorinv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_tensorsolve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_vecdot_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linalg_vector_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linspace_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linspace_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_linspace_tensor_overload_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_log10_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_log_softmax_with_dtype_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_logaddexp2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_logcumsumexp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_logcumsumexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_logdet_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_logical_not_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_logical_xor_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_logspace_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_logspace_tensor_overload_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_logsumexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_lu_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_mH_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_amax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_amin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_cumprod_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_cumprod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_cumsum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_cumsum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_fill_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_logaddexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_logsumexp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_softmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_std_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_std_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_masked_sum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_maximum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_meshgrid_list_of_tensors_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_meshgrid_variadic_tensors_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_mm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_mode_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_movedim_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_msort_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_mul_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_multinomial_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_mv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_mvlgamma_mvlgamma_p_3_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_mvlgamma_mvlgamma_p_5_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nanmean_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nanmean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nanmedian_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nanquantile_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_narrow_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_native_dropout_backward_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_ne_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_neg_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_new_empty_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_new_full_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_new_ones_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_adaptive_avg_pool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_batch_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_bilinear_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_binary_cross_entropy_with_logits_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_celu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_channel_shuffle_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_conv1d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_conv1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_conv2d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_conv2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_conv3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_conv_transpose1d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_conv_transpose1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_conv_transpose3d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_dropout3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_elu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_feature_alpha_dropout_with_train_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_feature_alpha_dropout_without_train_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_gaussian_nll_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_gelu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_grid_sample_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_hardshrink_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_hardswish_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_instance_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_l1_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_leaky_relu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_max_pool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_max_unpool1d_grad_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_max_unpool3d_grad_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_multi_margin_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_multilabel_margin_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_pad_reflect_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_pairwise_distance_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_pixel_shuffle_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_pixel_shuffle_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_pixel_unshuffle_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_relu6_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_relu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_rms_norm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_scaled_dot_product_attention_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_silu_complex_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_softsign_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_threshold_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_triplet_margin_loss_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_triplet_margin_with_distance_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nn_functional_upsample_bilinear_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nonzero_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nonzero_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nonzero_static_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_nonzero_static_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_norm_fro_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_norm_inf_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_normal_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_ones_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_ones_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_ones_like_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_ormqr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_ormqr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_outer_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_outer_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_pca_lowrank_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_pca_lowrank_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_permute_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_permute_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_pinverse_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_polar_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_polygamma_polygamma_n_1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_polygamma_polygamma_n_2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_polygamma_polygamma_n_3_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_polygamma_polygamma_n_4_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_prod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_qr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_quantile_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_rand_like_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_randint_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_randint_like_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_randn_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_randn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_real_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_reciprocal_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_renorm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_renorm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_repeat_interleave_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_reshape_as_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_reshape_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_resize_as__cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_resolve_neg_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_roll_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_roll_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_round_decimals_neg_3_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_rsqrt_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_scalar_tensor_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_scan_simple_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_scatter_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_scatter_reduce_amax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_scatter_reduce_prod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_searchsorted_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_select_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_select_scatter_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_sigmoid_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_signal_windows_bartlett_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_signal_windows_gaussian_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_signal_windows_general_hamming_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_signal_windows_hamming_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_signal_windows_kaiser_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_sinc_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_sinc_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_sinh_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_slice_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_slice_scatter_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_softmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_sort_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_airy_ai_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_bessel_j0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_bessel_j1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_bessel_y0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_chebyshev_polynomial_u_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_chebyshev_polynomial_w_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_hermite_polynomial_he_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_i1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_i1e_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_modified_bessel_i0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_shifted_chebyshev_polynomial_u_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_shifted_chebyshev_polynomial_w_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_spherical_bessel_j0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_xlog1py_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_special_zeta_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_split_list_args_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_split_list_args_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_square_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_squeeze_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_squeeze_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_squeeze_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_std_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_std_mean_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_std_mean_unbiased_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_std_unbiased_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_sub_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_sum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_sum_to_size_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_svd_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_svd_lowrank_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_t_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_t_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_take_along_dim_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_tan_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_tile_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_to_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_to_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_to_sparse_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_torch_ops_aten__safe_softmax_default_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_trapezoid_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_trapz_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_trapz_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_triangular_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_tril_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_triu_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_true_divide_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_true_divide_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_unbind_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_unbind_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_unflatten_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_uniform_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_unsafe_split_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_unsqueeze_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_var_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_var_unbiased_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_view_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_view_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_where_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_zeros_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_grad_zeros_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_H_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_NumpyCatCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_NumpyMulCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_NumpySplitCopyWithIntCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_NumpyTakeCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_NumpyViewCopyCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_T_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad___getitem___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad___radd___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad___rdiv___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad___rmatmul___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad___rmul___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad___rpow___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad__native_batch_norm_legit_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad__unsafe_masked_index_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad__unsafe_masked_index_put_accumulate_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_abs_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_abs_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_acos_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_addbmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_addcdiv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_addmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_addmv_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_addmv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_alias_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_amin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_argmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_argsort_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_argwhere_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_as_strided_partial_views_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_asin_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_asin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_atan_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_atan_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_atanh_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_atanh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_atleast_1d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_atleast_3d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_atleast_3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_baddbmm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_block_diag_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_broadcast_tensors_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_broadcast_to_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_cdist_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_cdouble_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_ceil_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_chalf_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_chalf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_char_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_cholesky_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_chunk_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_clamp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_clone_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_column_stack_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_combinations_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_cond_simple_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_conj_physical_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_contiguous_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_corrcoef_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_corrcoef_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_cross_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_cummin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_diag_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_diag_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_diagflat_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_diagflat_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_diagonal_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_diagonal_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_diagonal_scatter_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_diff_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_dot_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_dsplit_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_dsplit_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_empty_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_empty_permuted_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_empty_strided_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_erf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_erfinv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_exp2_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_expand_as_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_expand_as_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_expand_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_fft2_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_hfft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_hfftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_ifft2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_ifft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_ifftshift_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_ihfft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_ihfftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_irfft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_irfftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_rfft2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fft_rfftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_flatten_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_float_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_floor_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_floor_divide_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_fmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_ge_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_grid_sampler_2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_hsplit_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_hstack_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_i0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_index_put_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_index_reduce_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_index_reduce_prod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_inner_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_inner_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_isnan_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_isnan_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_isreal_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_istft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_item_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_jiterator_2inputs_2outputs_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_jiterator_binary_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_jiterator_binary_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_kron_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_ldexp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_lgamma_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_cholesky_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_cholesky_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_cond_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_cross_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_diagonal_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_eig_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_eigh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_householder_product_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_inv_ex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_ldl_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_lu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_lu_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_lu_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_multi_dot_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_norm_subgradients_at_zero_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_norm_subgradients_at_zero_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_pinv_hermitian_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_pinv_singular_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_svdvals_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_tensorinv_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_tensorsolve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linalg_vector_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linspace_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_linspace_tensor_overload_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_log1p_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_log1p_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_log_softmax_with_dtype_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_log_softmax_with_dtype_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_logical_or_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_logical_xor_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_logit_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_logspace_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_logspace_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_logspace_tensor_overload_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_logsumexp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_long_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_lu_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_lu_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_lu_unpack_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_mH_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_mT_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_masked_fill_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_masked_logsumexp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_masked_logsumexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_masked_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_masked_normalize_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_masked_prod_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_masked_softmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_masked_sum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_matmul_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_matrix_exp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_max_binary_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_mean_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_min_reduction_with_dim_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_movedim_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_mul_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_mvlgamma_mvlgamma_p_1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nan_to_num_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nanmean_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nanmedian_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_narrow_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_native_batch_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_native_dropout_backward_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_ne_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_neg_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_new_empty_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_new_empty_strided_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_new_ones_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_new_zeros_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_new_zeros_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_adaptive_max_pool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_adaptive_max_pool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_adaptive_max_pool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_alpha_dropout_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_avg_pool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_avg_pool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_batch_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_channel_shuffle_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_conv1d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_conv1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_conv3d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_conv_transpose1d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_conv_transpose1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_conv_transpose2d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_conv_transpose3d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_conv_transpose3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_cosine_embedding_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_cross_entropy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_embedding_bag_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_fractional_max_pool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_gaussian_nll_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_glu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_group_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_hardtanh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_hinge_embedding_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_huber_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_interpolate_linear_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_local_response_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_max_pool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_max_unpool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_max_unpool1d_grad_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_max_unpool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_multi_head_attention_forward_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_pad_reflect_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_pad_replicate_negative_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_pairwise_distance_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_pdist_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_pixel_shuffle_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_poisson_nll_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_relu6_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_relu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_rms_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_scaled_dot_product_attention_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_selu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nn_functional_softmin_with_dtype_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_nonzero_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_norm_nuc_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_ormqr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_outer_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_pca_lowrank_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_permute_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_permute_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_pinverse_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_pow_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_prod_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_put_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_qr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_qr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_rad2deg_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_rand_like_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_rand_like_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_randint_like_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_remainder_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_renorm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_repeat_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_repeat_interleave_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_reshape_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_resize_as__cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_resolve_neg_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_resolve_neg_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_roll_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_rot90_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_round_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_round_decimals_3_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_rsub_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_scalar_tensor_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_scatter_add_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_scatter_add_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_scatter_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_scatter_reduce_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_short_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sigmoid_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sign_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_signal_windows_general_cosine_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_signal_windows_general_hamming_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_signal_windows_hamming_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_signal_windows_kaiser_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_signal_windows_nuttall_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_signbit_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sin_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sinc_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_slice_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_slice_scatter_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sparse_sampled_addmm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sparse_sampled_addmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_bessel_j1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_bessel_y0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_bessel_y1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_chebyshev_polynomial_t_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_i0e_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_laguerre_polynomial_l_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_legendre_polynomial_p_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_scaled_modified_bessel_k1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_shifted_chebyshev_polynomial_u_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_shifted_chebyshev_polynomial_v_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_shifted_chebyshev_polynomial_w_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_spherical_bessel_j0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_special_zeta_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_split_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_split_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_split_list_args_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_split_with_sizes_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_split_with_sizes_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sqrt_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_square_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_square_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_squeeze_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_squeeze_multiple_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_stack_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_std_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_std_mean_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_std_mean_unbiased_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_std_unbiased_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_stft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sum_to_size_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_sum_to_size_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_t_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_t_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_take_along_dim_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_take_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_tan_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_tan_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_tensor_split_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_to_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_to_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_to_sparse_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_to_sparse_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_transpose_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_trapz_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_triu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_true_divide_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_unfold_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_unfold_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_unique_consecutive_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_unsafe_chunk_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_unsafe_chunk_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_var_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_var_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_var_mean_unbiased_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_var_unbiased_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_view_as_complex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_view_as_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_view_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_view_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_vsplit_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_zero__cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_zeros_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_fn_gradgrad_zeros_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_H_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_NumpyCatCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_NumpyMulScalarCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_NumpySplitCopyCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_NumpySplitCopyWithIntCustomOp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_T_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad___rmatmul___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad___rpow___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad___rsub___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad__chunk_cat_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad__native_batch_norm_legit_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad__segment_reduce_lengths_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad__segment_reduce_offsets_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad__unsafe_masked_index_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad__unsafe_masked_index_put_accumulate_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_abs_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_add_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_addbmm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_addcdiv_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_addcmul_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_addmm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_addmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_addmv_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_addmv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_addr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_alias_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_alias_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_amax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_as_strided_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_as_strided_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_asin_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_atan2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_atan_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_atanh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_atleast_3d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_bernoulli_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_block_diag_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_bmm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_bmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_broadcast_tensors_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_broadcast_to_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_byte_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_byte_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cartesian_prod_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cdist_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cdouble_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cfloat_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_char_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cholesky_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cholesky_inverse_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cholesky_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_chunk_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_clamp_max_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_clone_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_clone_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_column_stack_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_complex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_constant_pad_nd_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_corrcoef_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cos_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_count_nonzero_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cov_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cross_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cummax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cumsum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cumsum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_cumulative_trapezoid_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_deg2rad_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_diag_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_diagflat_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_diagonal_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_dist_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_div_floor_rounding_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_div_trunc_rounding_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_dot_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_double_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_dstack_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_einsum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_empty_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_empty_permuted_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_empty_strided_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_exp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_expand_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_expand_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_expand_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_eye_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_fft_fft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_fft_hfft2_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_fft_ifftshift_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_fft_ihfft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_fft_rfftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_fill_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_flatten_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_float_power_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_floor_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_fmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_fmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_frac_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_full_like_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_ge_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_geometric_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_histc_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_hsplit_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_hstack_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_hypot_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_i0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_index_add_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_index_fill_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_index_put_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_index_put_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_index_select_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_inner_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_int_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_isclose_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_isclose_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_isfinite_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_isin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_isinf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_isneginf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_isposinf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_istft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_jiterator_4inputs_with_extra_args_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_jiterator_4inputs_with_extra_args_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_jiterator_binary_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_jiterator_binary_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_jiterator_unary_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_kron_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_le_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_lerp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_lgamma_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_cholesky_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_cholesky_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_cholesky_ex_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_cholesky_ex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_cross_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_det_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_diagonal_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_eig_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_inv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_inv_ex_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_ldl_factor_ex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_ldl_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_lu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_lu_factor_ex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_multi_dot_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_norm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_pinv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_pinv_hermitian_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_pinv_singular_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_qr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_slogdet_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_solve_ex_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_solve_ex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_svd_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_tensorinv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_vander_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linalg_vander_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_linspace_tensor_overload_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_log1p_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_log_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_log_normal_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_log_softmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_log_softmax_with_dtype_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_logcumsumexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_logdet_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_logical_and_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_logical_not_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_logical_xor_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_logit_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_logspace_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_logspace_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_logspace_tensor_overload_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_long_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_lu_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_lu_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_lu_unpack_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_mH_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_mT_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_amin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_argmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_argmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_cumprod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_cumsum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_cumsum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_log_softmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_logaddexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_logsumexp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_logsumexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_scatter_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_masked_std_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_matmul_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_matrix_exp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_matrix_exp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_max_binary_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_max_reduction_no_dim_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_median_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_min_binary_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_movedim_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_mul_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_mul_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_multinomial_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_mv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_mvlgamma_mvlgamma_p_5_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nanquantile_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nansum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_narrow_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_native_batch_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_native_layer_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_ne_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_ne_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_new_empty_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_new_full_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_new_ones_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_adaptive_max_pool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_alpha_dropout_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_batch_norm_without_cudnn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_celu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_channel_shuffle_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_conv1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_conv2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_conv3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_conv_transpose1d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_conv_transpose1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_conv_transpose2d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_conv_transpose3d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_conv_transpose3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_cosine_embedding_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_elu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_embedding_bag_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_embedding_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_gaussian_nll_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_glu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_group_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_hardtanh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_interpolate_bicubic_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_l1_loss_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_l1_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_leaky_relu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_linear_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_max_pool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_max_pool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_max_pool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_max_unpool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_max_unpool2d_grad_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_max_unpool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_max_unpool3d_grad_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_mish_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_mse_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_multi_head_attention_forward_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_multi_margin_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_multilabel_margin_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_nll_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_normalize_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_pad_reflect_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_pad_replicate_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_pad_replicate_negative_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_pixel_shuffle_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_pixel_unshuffle_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_relu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_silu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_softmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_softmin_with_dtype_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_softmin_with_dtype_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_softsign_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_threshold_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_triplet_margin_loss_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_triplet_margin_with_distance_loss_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_triplet_margin_with_distance_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_unfold_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_unfold_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nn_functional_upsample_bilinear_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_nonzero_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_norm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_norm_inf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_normal_in_place_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_normal_in_place_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_ones_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_permute_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_permute_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_polar_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_polygamma_polygamma_n_1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_polygamma_polygamma_n_4_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_pow_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_prod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_put_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_qr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_quantile_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_randint_like_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_randn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_ravel_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_ravel_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_real_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_reciprocal_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_remainder_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_reshape_as_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_resize__cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_resize_as__cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_resolve_conj_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_resolve_neg_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_rot90_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_round_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_rsub_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_scatter_reduce_amax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_scatter_reduce_amin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_scatter_reduce_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_scatter_reduce_prod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_scatter_reduce_sum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_searchsorted_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_select_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_sigmoid_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_signal_windows_cosine_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_signal_windows_gaussian_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_sin_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_sin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_slice_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_slice_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_softmax_with_dtype_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_special_chebyshev_polynomial_v_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_special_chebyshev_polynomial_w_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_special_entr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_special_hermite_polynomial_he_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_special_laguerre_polynomial_l_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_special_log_ndtr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_special_modified_bessel_i1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_special_scaled_modified_bessel_k0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_special_scaled_modified_bessel_k1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_split_with_sizes_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_square_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_squeeze_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_std_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_std_mean_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_stft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_sub_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_sum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_sum_to_size_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_sum_to_size_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_svd_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_t_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_take_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_tanh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_tensordot_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_tile_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_torch_ops_aten__safe_softmax_default_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_transpose_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_trapz_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_triangular_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_triu_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_unbind_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_unflatten_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_unfold_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_unfold_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_uniform_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_unique_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_unsqueeze_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_unsqueeze_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_unsqueeze_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_var_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_var_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_var_unbiased_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_vdot_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_view_as_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_view_as_real_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_view_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_view_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_vsplit_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_vsplit_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_vstack_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_where_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_grad_zeros_like_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_H_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_T_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad___getitem___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad___radd___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad___rdiv___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad___rmatmul___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad___rmul___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad___rmul___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad___rsub___cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad___rsub___cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad__unsafe_masked_index_put_accumulate_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_abs_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_abs_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_acos_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_add_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_addbmm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_addbmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_addcmul_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_addcmul_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_addmm_decomposed_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_addmv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_amax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_angle_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_angle_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_arange_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_argwhere_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_argwhere_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_as_strided_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_as_strided_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_as_strided_partial_views_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_atan_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_atleast_1d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_atleast_2d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_atleast_3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_baddbmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_block_diag_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_block_diag_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_bmm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_broadcast_tensors_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_broadcast_to_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_byte_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_chalf_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_chalf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_char_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_char_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_cholesky_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_cholesky_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_cholesky_solve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_clamp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_combinations_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_conj_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_constant_pad_nd_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_contiguous_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_cos_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_count_nonzero_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_count_nonzero_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_cross_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_cumulative_trapezoid_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_diagonal_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_diagonal_scatter_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_diagonal_scatter_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_dist_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_div_floor_rounding_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_double_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_dstack_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_einsum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_empty_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_empty_like_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_eq_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_eq_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_erf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_erfc_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_erfinv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_exp2_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_expand_as_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_expand_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_expand_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_expm1_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_exponential_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_eye_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fft_fft2_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fft_fft_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fft_fft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fft_fftshift_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fft_hfft2_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fft_ifft_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fft_ifftshift_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fft_ifftshift_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fft_ihfftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fft_irfftn_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fill_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fill_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_flatten_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_float_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_float_power_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_fmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_frac_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_full_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_full_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_ge_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_geqrf_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_geqrf_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_heaviside_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_histc_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_hsplit_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_hstack_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_hypot_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_imag_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_index_add_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_index_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_index_fill_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_index_put_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_index_reduce_prod_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_index_select_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_index_select_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_isclose_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_isfinite_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_isreal_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_item_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_jiterator_2inputs_2outputs_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_jiterator_4inputs_with_extra_args_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_jiterator_binary_return_by_ref_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_jiterator_binary_return_by_ref_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_kron_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_ldexp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_cholesky_ex_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_cond_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_cond_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_cross_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_eig_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_eig_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_eigh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_eigvals_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_eigvalsh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_householder_product_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_inv_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_ldl_factor_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_lu_factor_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_lu_factor_ex_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_matrix_norm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_matrix_power_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_norm_subgradients_at_zero_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_pinv_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_pinv_hermitian_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_pinv_singular_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_solve_ex_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_solve_triangular_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_svd_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_tensorsolve_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_vander_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_vecdot_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linalg_vector_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linspace_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_linspace_tensor_overload_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_log_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_logcumsumexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_logdet_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_logdet_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_logical_or_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_logspace_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_logspace_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_logspace_tensor_overload_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_logspace_tensor_overload_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_logsumexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_lt_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_lu_solve_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_lu_unpack_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_mH_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_argmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_argmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_cumprod_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_cumsum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_cumsum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_log_softmax_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_logaddexp_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_logsumexp_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_median_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_prod_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_softmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_std_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_masked_sum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_max_pool2d_with_indices_backward_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_max_reduction_with_dim_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_mean_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_meshgrid_list_of_tensors_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_meshgrid_variadic_tensors_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_min_reduction_with_dim_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_minimum_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_mode_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_movedim_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_mul_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nan_to_num_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nansum_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_narrow_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_neg_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_new_empty_strided_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_new_ones_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nextafter_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_adaptive_avg_pool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_adaptive_avg_pool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_adaptive_avg_pool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_adaptive_max_pool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_adaptive_max_pool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_avg_pool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_avg_pool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_batch_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_binary_cross_entropy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_channel_shuffle_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_conv3d_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_conv_transpose1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_conv_transpose2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_ctc_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_embedding_bag_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_feature_alpha_dropout_without_train_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_hardshrink_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_hardsigmoid_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_hardtanh_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_hinge_embedding_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_interpolate_area_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_interpolate_bilinear_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_interpolate_linear_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_interpolate_nearest-exact_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_interpolate_nearest_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_linear_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_local_response_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_max_pool1d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_max_pool2d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_max_unpool3d_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_mish_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_mse_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_multi_head_attention_forward_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_multilabel_margin_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_normalize_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_pad_circular_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_pad_reflect_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_pad_reflect_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_pad_replicate_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_pad_replicate_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_pdist_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_prelu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_relu6_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_rms_norm_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_scaled_dot_product_attention_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_silu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_softmin_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_softshrink_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_softsign_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_tanhshrink_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_threshold_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_triplet_margin_loss_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_triplet_margin_with_distance_loss_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_unfold_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nn_functional_upsample_nearest_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_nonzero_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_norm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_norm_fro_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_normal_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_normal_number_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_ones_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_permute_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_permute_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_pinverse_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_polar_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_polygamma_polygamma_n_1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_polygamma_polygamma_n_4_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_put_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_qr_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_rad2deg_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_ravel_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_real_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_renorm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_repeat_interleave_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_resize__cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_resolve_neg_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_roll_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_rot90_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_rsub_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_select_scatter_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_short_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_sign_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_signal_windows_blackman_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_signal_windows_exponential_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_signal_windows_gaussian_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_signal_windows_general_cosine_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_signal_windows_hamming_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_signal_windows_hann_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_sinc_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_slice_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_slice_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_sort_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_sparse_mm_reduce_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_sparse_sampled_addmm_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_bessel_j1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_hermite_polynomial_h_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_i0e_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_i1e_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_laguerre_polynomial_l_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_modified_bessel_i0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_modified_bessel_i1_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_ndtr_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_polygamma_special_polygamma_n_0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_scaled_modified_bessel_k0_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_shifted_chebyshev_polynomial_t_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_shifted_chebyshev_polynomial_v_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_xlog1py_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_special_zeta_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_split_with_sizes_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_split_with_sizes_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_sqrt_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_std_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_std_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_std_mean_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_std_mean_unbiased_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_std_unbiased_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_svd_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_t_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_take_along_dim_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_take_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_tan_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_tan_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_tanh_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_tensor_split_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_tensordot_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_tile_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_tile_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_to_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_to_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_to_sparse_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_transpose_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_trapezoid_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_trapz_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_tril_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_triu_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_triu_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_true_divide_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_unbind_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_unbind_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_unbind_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_unfold_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_uniform_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_uniform_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_unsafe_split_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_unsqueeze_copy_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_unsqueeze_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_var_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_var_unbiased_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_view_as_cuda_complex128, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_view_as_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_view_copy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_xlogy_cuda_float64, test/test_ops_gradients.py::TestBwdGradientsCUDA::test_inplace_gradgrad_zeros_like_cuda_float64
2025-07-17T09:46:45.0262284Z 
2025-07-17T09:47:00.9016391Z 
2025-07-17T09:47:00.9017152Z test_ops 19/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_19.20_26e62770b8615b27_.log
2025-07-17T09:47:00.9534757Z Running 1726 items in this shard: test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs__conversions_cdouble_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_diag_embed_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_hypot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_igamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_unsqueeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_as_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_bernoulli_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_cummax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_diag_embed_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_diagonal_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_fliplr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_fmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_kthvalue_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_cholesky_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_cosine_embedding_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_fractional_max_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_margin_ranking_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_max_pool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_randint_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_randn_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_special_shifted_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_special_xlog1py_cuda_float32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_char_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_conj_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_hsplit_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_index_fill_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_ravel_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_roll_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes___ror___cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs__conversions_polar_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_amax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_chunk_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_fft_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fliplr_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_frac_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_isreal_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_lerp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_layer_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_relu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_special_xlog1py_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_unflatten_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_unfold_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_where_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_all_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_erf_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fft_ifftn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_histogram_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_int_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_lgamma_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_matrix_power_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_svd_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_lt_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_logaddexp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_softmax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_sum_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_min_reduction_no_dim_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_narrow_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_new_empty_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nextafter_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_adaptive_avg_pool1d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_adaptive_max_pool3d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_ctc_loss_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_dropout3d_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_instance_norm_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_softplus_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_rand_like_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_scatter_reduce_prod_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_slice_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_i0e_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_ndtr_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_split_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_triu_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_unfold_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_view_as_real_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_vstack_cuda, test/test_ops.py::TestCommonCUDA::test_errors_bitwise_or_cuda, test/test_ops.py::TestCommonCUDA::test_errors_dstack_cuda, test/test_ops.py::TestCommonCUDA::test_errors_fft_hfftn_cuda, test/test_ops.py::TestCommonCUDA::test_errors_fft_irfft_cuda, test/test_ops.py::TestCommonCUDA::test_errors_histogramdd_cuda, test/test_ops.py::TestCommonCUDA::test_errors_jiterator_binary_cuda, test/test_ops.py::TestCommonCUDA::test_errors_linalg_cross_cuda, test/test_ops.py::TestCommonCUDA::test_errors_minimum_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_adaptive_avg_pool2d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_gelu_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_max_pool2d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_signal_windows_general_cosine_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_randn_like_layout0_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_randn_like_layout3_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_sum_layout0_cuda, test/test_ops.py::TestCommonCUDA::test_errors_sparse_zeros_like_layout0_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_conj_physical_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_equal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_lgamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_eigh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_eigvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_ldl_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_round_decimals_neg_3_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_searchsorted_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_special_bessel_y1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_std_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_take_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_tensordot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_H_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices___rsub___cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_abs_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_as_strided_partial_views_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_atan2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_atleast_1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_atleast_3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_bool_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_ceil_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_conj_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_conj_physical_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_equal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_fftshift_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_irfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_half_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_inner_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_isneginf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_kthvalue_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_diagonal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_lu_factor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_qr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_argmax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_argmin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_masked_prod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_maximum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mode_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_movedim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mvlgamma_mvlgamma_p_3_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_native_batch_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_linear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pairwise_distance_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pairwise_distance_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_rrelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_polygamma_polygamma_n_0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_polygamma_polygamma_n_2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_randn_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_resize_as__cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_resolve_conj_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_signal_windows_hann_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_sort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_airy_ai_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_chebyshev_polynomial_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_shifted_chebyshev_polynomial_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_squeeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_squeeze_multiple_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_take_along_dim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_trapezoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_true_divide_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_var_mean_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_view_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_vsplit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_angle_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_as_strided_partial_views_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_as_strided_scatter_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_byte_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_clamp_min_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_deg2rad_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_eq_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_exp_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_masked_scatter_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_slice_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_bessel_y0_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_legendre_polynomial_p_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_modified_bessel_k1_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_shifted_chebyshev_polynomial_v_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_unsafe_split_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_zeros_like_cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples___rmod___cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addmv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_addr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_argmin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_argwhere_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_as_strided_partial_views_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_asin_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_asinh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atleast_2d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atleast_2d_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_bool_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cdouble_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cholesky_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_constant_pad_nd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cumprod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diag_embed_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_div_trunc_rounding_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_equal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_exp2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_hfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_hfft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_hfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_irfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fliplr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_flipud_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_frac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_full_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ge_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_reduce_prod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_select_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_isnan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_isreal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_jiterator_4inputs_with_extra_args_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_jiterator_binary_return_by_ref_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_eigh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_ldl_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lu_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lu_factor_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_median_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_meshgrid_variadic_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_min_reduction_no_dim_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_msort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_narrow_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_channel_shuffle_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_instance_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pad_constant_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_rms_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ravel_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_repeat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_repeat_interleave_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scatter_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scatter_reduce_prod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_signbit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_slice_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_chebyshev_polynomial_v_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sum_to_size_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_svd_lowrank_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_trunc_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_var_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_zeros_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_clone_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_diag_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_diff_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_jiterator_2inputs_2outputs_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_jiterator_4inputs_with_extra_args_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_linalg_vander_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_conv_transpose1d_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_group_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_rms_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_signal_windows_hamming_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_squeeze_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_unbind_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_out___rdiv___cuda_float32, test/test_ops.py::TestCommonCUDA::test_out___ror___cuda_int64, test/test_ops.py::TestCommonCUDA::test_out__refs__conversions_bfloat16_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs__conversions_complex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_atan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_bitwise_and_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out__refs_digamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_erf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_floor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_imag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out__refs_linalg_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_logical_not_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_mul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_mish_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_reshape_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_take_along_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_tensor_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_block_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_clamp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_clamp_min_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_fft_ihfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_isnan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_jiterator_binary_return_by_ref_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_jiterator_unary_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_det_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_matrix_rank_hermitian_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_qr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_masked_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_msort_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_silu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_softsign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_normal_in_place_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_normal_number_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_reciprocal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_repeat_interleave_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_acos_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_cumsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_erfc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_expand_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_i0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_cholesky_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_cholesky_ex_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_householder_product_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_pinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_log_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_log_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nanmean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_neg_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nn_functional_linear_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nn_functional_logsigmoid_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_sub_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_view_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_searchsorted_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_special_airy_ai_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_sqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_tensordot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_trace_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_triu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning___rand___cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_equal_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_fft_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_flatten_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_flip_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_alpha_dropout_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_nll_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_softshrink_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_erfcx_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_i1e_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_multigammaln_mvlgamma_p_5_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_transpose_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_view_as_complex_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_view_as_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_xlogy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_addmm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_bmm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_char_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_clamp_min_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_conj_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cosh_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_dist_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_inner_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_ldexp_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_lgamma_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_log1p_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_masked_fill_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_min_binary_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nanmedian_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_new_empty_strided_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_conv_transpose3d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_silu_complex_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_silu_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_norm_fro_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_rad2deg_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_rand_like_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_reciprocal_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_renorm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_special_ndtri_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_svd_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_unfold_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_view_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_vsplit_cuda, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atan2_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atan_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atanh_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_cos_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_digamma_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_div_no_rounding_mode_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erf_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_i0_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_lgamma_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_mvlgamma_mvlgamma_p_5_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_4_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sin_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sin_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_hermite_polynomial_h_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_hermite_polynomial_h_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_hermite_polynomial_he_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_t_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_v_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_w_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_zeta_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sqrt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_T_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bfloat16_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bfloat16_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_byte_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_float_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_half_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_int_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_polar_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_abs_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acosh_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcmul_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amax_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amin_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_partial_views_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_scatter_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_1d_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atleast_3d_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_shapes_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_tensors_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bucketize_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cat_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_clone_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_physical_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_physical_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cosh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_deg2rad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_scatter_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_scatter_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_trunc_rounding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dsplit_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dsplit_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_eq_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erf_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erfinv_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expm1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft2_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftshift_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfft2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfftn_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_divide_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_floor_divide_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fmax_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_fill_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isclose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isclose_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_item_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lgamma_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lgamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_cross_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_diagonal_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_diagonal_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linalg_vector_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_tensor_overload_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log10_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log1p_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log1p_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_and_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_xor_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_xor_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logspace_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_masked_fill_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_mean_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_list_of_tensors_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_list_of_tensors_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nan_to_num_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ne_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_neg_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_empty_strided_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_channel_shuffle_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_group_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_hardshrink_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_hinge_embedding_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_layer_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_log_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_margin_ranking_loss_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pdist_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_relu6_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmax_with_dtype_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_tanhshrink_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_threshold_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_triplet_margin_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_normal__in_place_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_normal_number_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_permute_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_prod_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rad2deg_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reciprocal_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_remainder_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_repeat_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_repeat_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_as_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_roll_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsqrt_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sin_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_bessel_j1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_erfcx_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i1_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_xlog1py_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sqrt_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_square_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_std_mean_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sub_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tensor_split_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tensor_split_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tensor_split_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tensor_split_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_to_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tril_indices_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_true_divide_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_true_divide_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_var_mean_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vdot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vstack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_xlogy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_add_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_arange_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_diag_embed_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_fft_hfft_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_fft_ifft_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_linspace_tensor_overload_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_mean_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_nn_functional_hardtanh_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_T_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bool_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cdouble_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cfloat_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cfloat_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_cfloat_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_char_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_complex_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_float_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_half_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_polar_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_short_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_short_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_short_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_short_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_abs_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_add_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_alias_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_allclose_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_amin_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_amin_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_any_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_scatter_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan2_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atanh_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_1d_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_3d_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_not_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_right_shift_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_block_diag_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_tensors_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_tensors_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cat_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ceil_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_min_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_column_stack_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_physical_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_constant_pad_nd_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_constant_pad_nd_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_contiguous_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cos_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cosh_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cosh_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_count_nonzero_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_deg2rad_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diag_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_scatter_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_no_rounding_mode_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dstack_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dstack_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_like_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_empty_strided_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expm1_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft2_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft2_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fft_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftn_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_fftn_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ihfft_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft2_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft2_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft2_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft2_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fill_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_flatten_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fliplr_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_float_power_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_float_power_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmin_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fmin_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_frac_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_gt_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hsplit_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isfinite_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isnan_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isneginf_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isneginf_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isreal_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_le_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_diagonal_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_tensor_overload_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_and_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_not_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_not_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logspace_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logspace_tensor_overload_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logspace_tensor_overload_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logsumexp_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logsumexp_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logsumexp_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_masked_fill_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_minimum_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mul_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_native_layer_norm_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_native_layer_norm_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ne_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_neg_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_full_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_zeros_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_zeros_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_zeros_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hardtanh_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_huber_loss_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_l1_loss_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_leaky_relu_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pdist_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_shuffle_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_shuffle_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmax_with_dtype_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmin_with_dtype_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softplus_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_threshold_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_norm_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_positive_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_pow_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rad2deg_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rot90_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsqrt_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsqrt_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsqrt_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rsub_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_select_scatter_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sgn_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sigmoid_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sin_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinc_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinh_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_softmax_with_dtype_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_bessel_j0_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_erfcx_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_i1e_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_ndtr_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_softmax_with_dtype_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_logit_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_multigammaln_mvlgamma_p_5_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_softmax_with_dtype_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_copy_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_copy_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_squeeze_multiple_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sub_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_to_size_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sum_to_size_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tan_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tan_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tanh_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tanh_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unbind_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unflatten_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_copy_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unsqueeze_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_var_mean_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_complex_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vsplit_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vstack_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vstack_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vstack_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_where_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_xlogy_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_T_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bfloat16_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bfloat16_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_bool_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_double_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_double_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_int_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_long_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_short_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_abs_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_acosh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_all_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_amax_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_amin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_partial_views_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_partial_views_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_scatter_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asin_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asinh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_3d_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_3d_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_or_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_block_diag_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_tensors_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_tensors_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clone_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_column_stack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_copysign_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumsum_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumsum_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_deg2rad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_no_rounding_mode_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_div_trunc_rounding_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eq_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfinv_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfinv_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_as_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expm1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_float8_e5m2fnuz, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_fftshift_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftn_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ihfft_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_irfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fill_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fill_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flip_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmax_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmin_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ge_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_i0_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_i0_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_i0_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_igamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_fill_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_select_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isinf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isnan_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isneginf_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isreal_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_item_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lcm_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_cross_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_cross_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_diagonal_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_matrix_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_matrix_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_norm_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log10_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logaddexp_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_and_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_xor_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lt_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_meshgrid_variadic_tensors_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_minimum_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_movedim_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mul_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mul_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ne_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_strided_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_ones_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_celu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_channel_shuffle_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_channel_shuffle_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hardshrink_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_margin_ranking_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_shuffle_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_shuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_shuffle_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_unshuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_relu6_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_selu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_selu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmax_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmin_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmin_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_softmin_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_threshold_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_normal__in_place_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_prod_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_real_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_real_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_remainder_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_remainder_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_renorm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_repeat_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_as_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_as_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_as_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_roll_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rot90_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_round_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsqrt_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsqrt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sigmoid_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_softmax_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_bessel_j1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_i1e_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_ndtr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_multigammaln_mvlgamma_p_1_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtr_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtri_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_ndtri_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_softmax_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_zeta_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_square_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_square_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_std_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_std_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sub_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sub_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tan_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_to_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trace_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tril_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_triu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trunc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unbind_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_var_mean_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_var_mean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_xlogy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_zeros_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_zeros_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bfloat16_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_byte_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_byte_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cfloat_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_chalf_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_chalf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_char_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_int_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_long_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acosh_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_add_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_addr_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_alias_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amax_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_partial_views_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_asinh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_2d_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_2d_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atleast_3d_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_left_shift_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_block_diag_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_block_diag_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_to_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_conj_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cosh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cosh_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumprod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumsum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diagonal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_no_rounding_mode_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_no_rounding_mode_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_div_no_rounding_mode_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dot_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dsplit_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dstack_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dstack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_dstack_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_like_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eq_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_equal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eye_cuda_float8_e4m3fn, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_fftshift_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_hfftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifftn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifftshift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfftn_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_irfft_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fliplr_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flipud_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_divide_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmod_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_geometric_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hsplit_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_hstack_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isclose_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isinf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isposinf_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_item_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_item_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lcm_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lgamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_diagonal_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_norm_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_tensor_overload_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_tensor_overload_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_and_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_not_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_or_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_or_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logspace_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logsumexp_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logsumexp_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_masked_fill_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_maximum_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nan_to_num_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_neg_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_neg_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_strided_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nextafter_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_celu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_channel_shuffle_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_gelu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_hinge_embedding_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_log_softmax_with_dtype_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_log_softmax_with_dtype_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_poisson_nll_loss_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_relu_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_smooth_l1_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmin_with_dtype_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softshrink_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_tanhshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_threshold_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_triplet_margin_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_permute_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_positive_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_prod_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rad2deg_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_randn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_real_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_real_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reciprocal_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_remainder_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_as_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_roll_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_round_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsqrt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinc_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_bessel_j1_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_entr_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_erfcx_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i0e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1e_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_ndtr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_logit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_3_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtr_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtri_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_softmax_with_dtype_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sqrt_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sqrt_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_square_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_square_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tensor_split_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trace_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trace_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_transpose_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_true_divide_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_trunc_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unbind_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unflatten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vsplit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_vstack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_zeros_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___rmatmul___cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_bool_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_chalf_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_combinations_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_conj_physical_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_digamma_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_double_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_equal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_fft2_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_hfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_irfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_rfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_frexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_full_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_half_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_index_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_isneginf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_isreal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_kron_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_lerp_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_lu_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_multi_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_svdvals_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_log_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_median_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_cosine_embedding_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_instance_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_multi_head_attention_forward_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_softplus_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_norm_fro_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_ones_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_rand_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_reshape_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_scatter_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_scatter_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_special_i1e_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_split_with_sizes_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sqrt_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_std_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_svd_lowrank_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_tril_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unflatten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unfold_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_uniform_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_var_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_view_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_view_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward___rsub___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_acosh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_addmm_decomposed_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_as_strided_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_as_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_atanh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_atleast_1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_bmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_broadcast_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_cholesky_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_chunk_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_diagonal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_fftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_ihfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_float_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_hypot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_householder_product_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_pinv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_vecdot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_logit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_logsumexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_hardtanh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_instance_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_mish_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_selu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_upsample_bilinear_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_rad2deg_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_reshape_as_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_sin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_special_i1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_t_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_topk_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_atan2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_char_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_dsplit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_empty_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_empty_strided_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_irfft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_floor_divide_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_hstack_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_isreal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_mT_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_matrix_exp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_new_ones_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_new_zeros_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_interpolate_nearest-exact_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_logsigmoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_relu6_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_norm_nuc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_ravel_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_remainder_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_sign_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_slice_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_bessel_y0_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_i1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_log_ndtr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_modified_bessel_k1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_ndtr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_special_zeta_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_std_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_topk_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_unfold_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad___rmod___cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_abs_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_any_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_as_strided_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_cartesian_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_diff_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_frac_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_full_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_isposinf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_jiterator_unary_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_eigvalsh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_log_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_logical_not_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_argmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_min_binary_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_narrow_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_kl_div_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_selu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_ormqr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_randn_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_resize_as__cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_tan_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_transpose_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_cat_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_count_nonzero_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_dot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_einsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_expand_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_rfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_flip_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_geqrf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_index_put_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_inv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_ldl_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_lu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_lu_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nanmedian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nanquantile_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_embedding_bag_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_max_unpool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_pad_replicate_negative_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_triplet_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_normal_number_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_resize_as__cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_special_shifted_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_unfold_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_unique_consecutive_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_vdot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_zeros_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_acos_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_all_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_argwhere_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_atan2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_baddbmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_broadcast_to_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_chalf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_cumulative_trapezoid_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_diagonal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_fftshift_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_ifft2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_geometric_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_isinf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_diagonal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_eigh_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_eigvals_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_logdet_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_narrow_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nextafter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_cross_entropy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_elu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_fractional_max_pool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_max_pool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_max_unpool3d_grad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_normal_number_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_ones_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_rsub_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_slice_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_squeeze_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_squeeze_multiple_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_topk_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_var_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_var_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_view_as_complex_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_cdouble_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_chalf_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_half_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_block_diag_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_diagonal_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_fft2_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_hfft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_linalg_vector_norm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_log10_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_new_zeros_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_nn_functional_pixel_shuffle_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_roll_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_rot90_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_unfold_copy_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_allclose_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_bfloat16_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_float_power_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_imag_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_cond_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_matrix_power_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_norm_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_norm_subgradients_at_zero_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_svd_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_lu_solve_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_new_empty_strided_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nn_functional_conv_transpose1d_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_std_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_std_mean_unbiased_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_svd_lowrank_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_take_along_dim_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_take_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_trace_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_var_mean_unbiased_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs__conversions_float_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_exp2_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_fft_ifftn_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_flatten_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_log10_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_log1p_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_nn_functional_channel_shuffle_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_nn_functional_triplet_margin_loss_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_reshape_as_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_unfold_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_var_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_alias_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_baddbmm_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_conj_physical_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_equal_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_expm1_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fft_ifft_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_flipud_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_full_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_gradient_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_jiterator_binary_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_jiterator_unary_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_tensorinv_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_logsumexp_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_linear_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_pixel_shuffle_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_tanhshrink_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_ones_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_permute_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_resolve_conj_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_resolve_neg_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_squeeze_multiple_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_zeros_like_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_addcdiv_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_addcmul_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_constant_pad_nd_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_dot_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_exp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_fft_rfft2_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_frexp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_heaviside_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_i0_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_index_fill_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_isreal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_masked_fill_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_meshgrid_list_of_tensors_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_meshgrid_variadic_tensors_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_gelu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_mse_loss_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_permute_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_sigmoid_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_special_i1e_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_triu_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_unsqueeze_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_var_mean_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_all_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_argmax_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_atan_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_char_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cholesky_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cov_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_digamma_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_einsum_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fft_fft2_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fft_hfft2_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fft_ifftshift_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fft_ihfft2_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_heaviside_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_igamma_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_index_put_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_jiterator_binary_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_ldexp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_normalize_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_scatter_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_mul_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_multinomial_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_native_layer_norm_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_cosine_similarity_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_max_unpool1d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_modified_bessel_i1_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_modified_bessel_k1_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_stack_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake___ror___cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast___rdiv___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_bincount_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_block_diag_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_broadcast_shapes_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_char_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cov_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cross_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_exponential_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_ihfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_frexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_isinf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_jiterator_4inputs_with_extra_args_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_ldexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_logaddexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_long_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_lu_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_max_binary_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_new_full_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_adaptive_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_gelu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_rot90_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_signal_windows_general_hamming_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_i1e_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_modified_bessel_k0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_tan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_transpose_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_view_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_bucketize_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_cholesky_inverse_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_H_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_acosh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_addmv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_cholesky_solve_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_clone_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_deg2rad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_exp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_fftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_hfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_ifft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_frexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_half_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_index_reduce_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_lerp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_inv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_masked_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_masked_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_max_reduction_no_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_mm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_narrow_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_conv1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_interpolate_bilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_interpolate_trilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_logsigmoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_max_unpool3d_grad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_multi_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_repeat_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_sign_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_special_erfcx_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_split_list_args_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_std_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_transpose_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_transpose_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_trapezoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_unfold_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_var_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp___radd___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp___rdiv___cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp__softmax_backward_data_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp__upsample_bilinear2d_aa_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_atleast_3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_bernoulli_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_copysign_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_irfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_rfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fft_rfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_float_power_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_floor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_half_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_index_reduce_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_eigh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_lstsq_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_matrix_power_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_log2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_conv3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_embedding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_hardsigmoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_interpolate_linear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_max_unpool3d_grad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_normalize_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_pdist_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_unfold_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_reciprocal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_roll_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_round_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_special_polygamma_special_polygamma_n_0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_special_xlog1py_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_to_sparse_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_view_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_hfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_irfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_index_reduce_amin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_index_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_lerp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_eigvalsh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_qr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_slogdet_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_log_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_mvlgamma_mvlgamma_p_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_new_empty_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_conv_transpose2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_dropout_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_softplus_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_norm_nuc_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_permute_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_remainder_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_renorm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_rot90_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_round_decimals_neg_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_scalar_tensor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_signal_windows_cosine_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_signal_windows_exponential_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_erfcx_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_scaled_modified_bessel_k0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_split_with_sizes_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_squeeze_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_torch_ops_aten__flash_attention_forward_cuda_float16, test/test_ops.py::TestFakeTensorCUDA::test_fake_triu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_unfold_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_view_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops__batch_norm_with_update_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_abs_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_addbmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_as_strided_partial_views_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_column_stack_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_erf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_floor_divide_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fmin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_frac_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_det_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_multi_dot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_svdvals_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_vecdot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_logical_and_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_median_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_mode_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_narrow_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_new_full_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_adaptive_max_pool1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_cosine_embedding_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_gaussian_nll_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_hardsigmoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_pdist_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_normal_in_place_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_permute_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_round_decimals_0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_signal_windows_gaussian_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_sort_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_bessel_y1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_modified_bessel_i0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_shifted_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_square_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_tan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_tensor_split_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_torch_ops_aten__safe_softmax_default_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_unique_consecutive_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_view_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_zeros_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_linspace_tensor_overload_cuda_bfloat16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_tensor_overload_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_tensor_overload_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_ones_cuda_bool, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_full_cuda_float16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_linspace_tensor_overload_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_bfloat16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_complex64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_int32, test/test_ops.py::TestTagsCUDA::test_tags__refs_acos_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_addcmul_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_diag_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fill_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_flip_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_pdist_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_ravel_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_bessel_j0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_erfcx_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_sum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_acos_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_as_strided_scatter_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_atleast_3d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_cholesky_solve_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_combinations_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_conj_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_diag_embed_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_dot_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_fftn_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_hfft2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_frexp_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_index_reduce_mean_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_kron_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_eigh_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_logit_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_minimum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_adaptive_avg_pool3d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_avg_pool3d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_cosine_embedding_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_l1_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_one_hot_cuda_int64, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_soft_margin_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_upsample_nearest_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nonzero_static_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_pow_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_quantile_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_scatter_reduce_amax_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_scatter_reduce_sum_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_searchsorted_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_sigmoid_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_modified_bessel_i0_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_sum_to_size_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_take_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_var_unbiased_cuda_float32
2025-07-17T09:47:00.9986520Z 
2025-07-17T09:47:39.3599817Z 
2025-07-17T09:47:39.3600874Z test_ops 20/20 was successful, full logs can be found in artifacts with path test/test-reports/test_ops_20.20_4b16a4eeaf2ae9a4_.log
2025-07-17T09:47:39.4088838Z Running 1629 items in this shard: test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs__conversions_polar_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_exponential_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_log_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_alpha_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_true_divide_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__refs_view_as_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu__unsafe_masked_index_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_as_strided_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_block_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_einsum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_flip_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_linalg_inv_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_median_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_min_reduction_no_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_narrow_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_avg_pool1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_linear_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_nonzero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_norm_nuc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_ones_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_repeat_interleave_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_resolve_neg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_select_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_split_cuda_float32, test/test_ops.py::TestCommonCUDA::test_compare_cpu_view_cuda_float32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_conj_physical_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_diagonal_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_fft_irfft_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_hstack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_mul_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_permute_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_true_divide_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_unbind_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_complex_half_reference_testing_unsafe_split_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_dtypes___rdiv___cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__batch_norm_with_update_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_as_strided_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_asin_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_clamp_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_contiguous_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_count_nonzero_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_div_no_rounding_mode_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_eye_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_ihfftn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_fft_irfftn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_flipud_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_imag_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_isclose_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_narrow_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_new_ones_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_relu6_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_softmax_with_dtype_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_softmin_with_dtype_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_nn_functional_threshold_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_rad2deg_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_randn_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_rsub_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_stft_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_unbind_copy_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_view_as_complex_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_view_as_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__refs_zeros_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes__unsafe_masked_index_put_accumulate_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_angle_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_diagflat_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_erfinv_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_fft_rfft2_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_floor_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_gcd_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_igamma_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_ldl_solve_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linalg_pinv_singular_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_linspace_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_lu_unpack_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_masked_amax_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_mvlgamma_mvlgamma_p_3_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nan_to_num_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_nn_functional_embedding_bag_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_resize_as__cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_signal_windows_cosine_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_signal_windows_exponential_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_signal_windows_gaussian_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_special_hermite_polynomial_he_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_take_along_dim_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_tan_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_trace_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_triu_indices_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_unsafe_chunk_cuda, test/test_ops.py::TestCommonCUDA::test_dtypes_zeros_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_conv1d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_conv3d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_nn_functional_max_pool3d_cuda, test/test_ops.py::TestCommonCUDA::test_errors_polar_cuda, test/test_ops.py::TestCommonCUDA::test_errors_special_laguerre_polynomial_l_cuda, test/test_ops.py::TestCommonCUDA::test_errors_special_shifted_chebyshev_polynomial_v_cuda, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_acosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_copysign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_cosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_dstack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_expand_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_float_power_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_index_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_linalg_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_log2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_log_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_nn_functional_softshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_reciprocal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_stack_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_var_cuda_float32, test/test_ops.py::TestCommonCUDA::test_meta_consistency_out_dtype_mismatch_zeros_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices___rmod___cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices___rsub___cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_acosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_add_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_addmm_decomposed_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cat_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cat_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_cholesky_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_chunk_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_clamp_max_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_contiguous_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_contiguous_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_copysign_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_exponential_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_eye_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_irfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_irfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_fft_rfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_float_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_floor_divide_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_gather_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_hsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_igammac_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_index_reduce_amin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_kthvalue_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_linalg_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_logical_and_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_long_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_matrix_exp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_mul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_neg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_hardtanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_huber_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_max_unpool3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_normalize_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pad_circular_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pad_circular_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pad_replicate_negative_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pixel_shuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_pixel_unshuffle_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_nn_functional_triplet_margin_with_distance_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_pca_lowrank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_rad2deg_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_reciprocal_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_select_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_signal_windows_blackman_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_special_zeta_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_std_mean_unbiased_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_topk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_multiple_devices_tril_indices_cuda_int64, test/test_ops.py::TestCommonCUDA::test_multiple_devices_where_cuda_int64, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values__unsafe_masked_index_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_amax_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_atanh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_cummax_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_expand_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_flatten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_jiterator_4inputs_with_extra_args_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_log2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_masked_sum_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_movedim_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_narrow_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_scalar_tensor_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_sinh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_airy_ai_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_scaled_modified_bessel_k1_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_special_shifted_chebyshev_polynomial_t_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_tanh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_non_standard_bool_values_tril_cuda_bool, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples__chunk_cat_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_argmax_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_argwhere_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_atanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_block_diag_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cat_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cov_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_cross_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diagflat_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_diff_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_erfc_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_expand_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_expm1_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_fftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ifftshift_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ifftshift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_fft_ihfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_full_like_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_geqrf_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_fill_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_index_put_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_le_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_cond_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_lu_factor_ex_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_multi_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_linalg_slogdet_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log10_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log1p_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_log_normal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_logit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_lu_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mT_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_masked_var_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ne_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_new_empty_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_conv3d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_mse_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pairwise_distance_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_nn_functional_pairwise_distance_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_prod_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_randint_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_ravel_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_scatter_reduce_sum_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_select_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_signal_windows_hamming_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_bessel_j0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_modified_bessel_i0_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_modified_bessel_k1_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_shifted_chebyshev_polynomial_v_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_special_xlog1py_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sqrt_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_squeeze_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_sum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_tril_indices_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_true_divide_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unsafe_chunk_cuda_float32, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_unsafe_split_cuda_int64, test/test_ops.py::TestCommonCUDA::test_noncontiguous_samples_where_cuda_float32, test/test_ops.py::TestCommonCUDA::test_numpy_ref_nn_functional_conv_transpose2d_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_ravel_cuda_int64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_signal_windows_hann_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_signal_windows_nuttall_cuda_float64, test/test_ops.py::TestCommonCUDA::test_numpy_ref_tile_cuda_float64, test/test_ops.py::TestCommonCUDA::test_out__refs_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_as_strided_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_as_strided_partial_views_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_atleast_1d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_clone_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_ihfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_fft_rfft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_isclose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_linalg_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_logical_and_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_logical_or_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_logical_xor_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_logsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_maximum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_round_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_sign_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_i1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_log_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_special_ndtr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_tanh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out__refs_vdot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_addbmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_argmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_bitwise_left_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_out_diagonal_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_equal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_erfinv_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_hsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_index_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_inner_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_integral_dtype__refs_sum_cuda_int16, test/test_ops.py::TestCommonCUDA::test_out_linalg_cond_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_linalg_eigh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nanmean_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nanquantile_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_new_empty_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_adaptive_avg_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_embedding_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_local_response_norm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_max_pool2d_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_smooth_l1_loss_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_nn_functional_unfold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_permute_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_add_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_alias_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_atan_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_exp2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_fft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_ifft_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_fft_irfftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_hypot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_kthvalue_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_cross_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_inv_ex_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_matrix_power_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_multi_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_pinv_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_qr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_slogdet_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linalg_solve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_linspace_tensor_overload_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_minimum_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_nansum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_polygamma_polygamma_n_2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_round_decimals_0_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_take_along_dim_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_out_requires_grad_error_tril_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_signal_windows_bartlett_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_signal_windows_general_hamming_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_sinh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_softmax_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_split_list_args_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_t_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_torch__scaled_mm_cuda_float8_e4m3fn, test/test_ops.py::TestCommonCUDA::test_out_trapz_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_unbind_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_view_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_out_warning__refs__conversions_half_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_bitwise_left_shift_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_diagonal_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_div_no_rounding_mode_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_fft_hfftn_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_flipud_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_linalg_vector_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_logical_and_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_narrow_copy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_ne_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_nn_functional_margin_ranking_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_permute_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_rsub_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_sinc_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_i0e_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_multigammaln_mvlgamma_p_1_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_special_xlog1py_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_take_along_dim_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_vdot_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning__refs_vstack_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_addbmm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_all_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_atleast_2d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_bitwise_xor_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_byte_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_corrcoef_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_cummin_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_diagonal_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_empty_like_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_fft_hfft2_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_full_like_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_histogram_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_lstsq_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_slogdet_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_linalg_solve_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_log_softmax_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_mT_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_new_empty_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_avg_pool3d_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_binary_cross_entropy_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_group_norm_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_huber_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_multilabel_margin_loss_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_pad_constant_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_nn_functional_pixel_unshuffle_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_polygamma_polygamma_n_0_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_softmax_with_dtype_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_sparse_mm_reduce_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_torch_ops_aten__flash_attention_forward_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_tril_cuda, test/test_ops.py::TestCommonCUDA::test_out_warning_unbind_cuda, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_asin_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_asinh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_atan_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_cosh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_deg2rad_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_digamma_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_div_no_rounding_mode_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_div_no_rounding_mode_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_div_no_rounding_mode_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erf_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erfc_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_erfc_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_exp2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_float_power_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_lgamma_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log10_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log1p_cuda_int32, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_log1p_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_masked_var_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_masked_var_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_polygamma_polygamma_n_3_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_reciprocal_cuda_int8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_rsqrt_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sigmoid_cuda_bool, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_sinh_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_t_cuda_int16, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_chebyshev_polynomial_v_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_shifted_chebyshev_polynomial_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_promotes_int_to_float_special_xlog1py_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_T_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_bool_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_byte_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cdouble_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cdouble_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_cdouble_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_chalf_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_double_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_float_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_int_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_int_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_int_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs__conversions_long_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_acosh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addcmul_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_addr_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_alias_copy_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_amin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_any_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_as_strided_scatter_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atan2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_atanh_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_bitwise_or_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_broadcast_to_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_chunk_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_column_stack_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_conj_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_constant_pad_nd_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_count_nonzero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_count_nonzero_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_cumprod_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diag_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_diagonal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_floor_rounding_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_no_rounding_mode_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_no_rounding_mode_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_div_trunc_rounding_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dot_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dsplit_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_dstack_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_empty_like_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_erfc_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_exp2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_expand_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_fftn_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_hfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftshift_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ifftshift_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfft2_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_ihfft_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfft2_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_irfftn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fft_rfft_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flip_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_fliplr_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_flipud_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ge_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_geometric_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_geometric_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_heaviside_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_heaviside_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hsplit_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hstack_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_hypot_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_add_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_fill_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_index_select_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isnan_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_isreal_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lcm_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_lgamma_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_linspace_tensor_overload_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_log2_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_xor_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_logical_xor_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_masked_fill_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_list_of_tensors_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_meshgrid_variadic_tensors_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nan_to_num_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_narrow_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_ones_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_new_zeros_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nextafter_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nextafter_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_gelu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_glu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_l1_loss_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_leaky_relu_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pairwise_distance_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_pixel_unshuffle_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_poisson_nll_loss_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_prelu_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_relu6_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_relu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_softmax_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_tanhshrink_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_nn_functional_threshold_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_pow_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_ravel_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_repeat_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_repeat_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_reshape_as_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_roll_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rot90_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_rsqrt_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sgn_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_signbit_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinc_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_sinh_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_bessel_j0_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_erfcx_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_i0e_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_log_softmax_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_logit_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_multigammaln_mvlgamma_p_1_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_special_ndtri_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_split_with_sizes_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_square_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_squeeze_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_stack_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_std_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_t_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tan_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tan_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_tanh_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_to_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_to_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_transpose_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_triu_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unbind_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unflatten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unfold_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_unsqueeze_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_var_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_view_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_vstack_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_where_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_where_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_xlogy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_xlogy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref__refs_zeros_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_as_strided_scatter_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_bitwise_left_shift_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_float_power_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_native_layer_norm_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_errors__refs_normal__in_place_cuda, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bfloat16_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_bool_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_byte_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_chalf_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_chalf_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_half_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_short_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs__conversions_short_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_abs_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acos_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_acosh_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_add_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_add_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_addcmul_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_amax_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_as_strided_scatter_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asin_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_asinh_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan2_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan2_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atan2_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atanh_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atanh_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_atleast_2d_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_bitwise_and_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_to_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_broadcast_to_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clamp_min_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_clone_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_column_stack_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_column_stack_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_column_stack_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_column_stack_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_conj_physical_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_cumsum_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_diagonal_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_floor_rounding_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_div_trunc_rounding_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dot_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_dsplit_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_eq_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_equal_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_exp_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_as_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_copy_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_expand_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_ifft2_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfft_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_irfftn_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_fft_rfftn_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_divide_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_divide_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_floor_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_frac_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_gcd_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_ge_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_gt_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_heaviside_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_hypot_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_add_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_add_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_index_select_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isclose_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isclose_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isclose_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isfinite_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_isfinite_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_item_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_item_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lgamma_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lgamma_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_vector_norm_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linalg_vector_norm_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_linspace_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log10_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log2_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_log_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logical_or_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logspace_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_logspace_tensor_overload_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_lt_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_masked_fill_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_masked_fill_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_maximum_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mul_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_mul_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nan_to_num_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_empty_strided_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_new_zeros_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_alpha_dropout_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_channel_shuffle_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_dropout_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_group_norm_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_hinge_embedding_loss_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_mish_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_mse_loss_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_nll_loss_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_pixel_shuffle_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_poisson_nll_loss_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_relu_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_softmax_with_dtype_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_tanhshrink_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_tanhshrink_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_tanhshrink_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_tanhshrink_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_threshold_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_nn_functional_triplet_margin_loss_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_norm_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_normal_number_mean_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_copy_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_permute_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_pow_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_prod_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_real_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reciprocal_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reciprocal_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_reshape_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_roll_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_roll_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_rot90_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_select_scatter_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sign_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_signbit_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_signbit_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sinh_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_entr_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_entr_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_erfcx_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_log_softmax_with_dtype_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_logit_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_zeta_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_special_zeta_executor_aten_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_sqrt_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_std_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_std_mean_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_t_executor_aten_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_take_along_dim_executor_aten_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tanh_executor_aten_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tanh_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_to_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_trace_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_executor_aten_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_transpose_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_tril_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_triu_executor_aten_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unflatten_executor_aten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unflatten_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_copy_executor_aten_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_copy_executor_aten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_unfold_copy_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vdot_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_as_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_view_executor_aten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vsplit_executor_aten_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_vstack_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_where_executor_aten_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_executor__refs_xlogy_executor_aten_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_byte_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_chalf_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_char_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_long_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs__conversions_short_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_abs_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_add_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_addr_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_alias_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_all_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_all_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_copy_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_as_strided_scatter_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asin_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_asinh_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atan_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atanh_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_1d_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_atleast_1d_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bitwise_right_shift_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_block_diag_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_tensors_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_to_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_to_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_broadcast_to_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_bucketize_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_chunk_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_chunk_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_clone_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_column_stack_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_physical_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_conj_physical_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_constant_pad_nd_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_contiguous_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cos_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cos_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_count_nonzero_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumsum_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_cumsum_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_deg2rad_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diag_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_scatter_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_scatter_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_diagonal_scatter_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_dstack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_like_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_empty_like_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_erfc_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exp2_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_as_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expand_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expm1_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_expm1_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_exponential_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_eye_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftn_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftshift_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_ifftshift_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fft_rfftn_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fill_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flatten_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_flatten_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_floor_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmax_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmin_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_fmod_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ge_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_geometric_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_heaviside_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_hsplit_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_i0_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_igammac_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_add_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_add_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_index_select_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isclose_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_isneginf_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_item_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lcm_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lerp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_diagonal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_linalg_svd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log10_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_softmax_with_dtype_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_log_softmax_with_dtype_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logical_or_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logspace_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_logsumexp_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_lt_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_masked_fill_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mean_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_mul_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_narrow_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_empty_strided_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_ones_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_ones_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_new_zeros_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_dropout_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_hardtanh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_layer_norm_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_layer_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_mse_loss_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pairwise_distance_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_pixel_unshuffle_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_relu6_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_nn_functional_triplet_margin_loss_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_ones_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_permute_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_prod_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rad2deg_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reciprocal_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_repeat_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_repeat_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_as_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_as_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_reshape_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_roll_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rot90_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rot90_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rot90_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsqrt_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_rsqrt_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sgn_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sin_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sin_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinc_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sinh_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_softmax_with_dtype_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_ndtr_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_special_log_softmax_with_dtype_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sqrt_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_square_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_square_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_squeeze_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_stack_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_std_mean_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_std_mean_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_sum_to_size_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_take_along_dim_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_tanh_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_trace_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_transpose_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_triu_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_true_divide_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unfold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_unsqueeze_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_as_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_view_as_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vstack_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_meta__refs_vstack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bfloat16_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_bool_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cdouble_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_cfloat_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_chalf_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_double_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_double_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_float_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs__conversions_short_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_abs_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_abs_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acos_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_acosh_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_add_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_alias_copy_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_allclose_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_amin_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_any_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_arange_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_partial_views_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_partial_views_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_as_strided_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_atan_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bitwise_or_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_tensors_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_broadcast_to_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_bucketize_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cauchy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ceil_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_chunk_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_chunk_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_max_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_max_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clamp_max_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_clone_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_column_stack_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_copysign_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cos_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_cumsum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_deg2rad_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_diag_embed_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_digamma_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_digamma_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_empty_like_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_eq_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_equal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erf_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erf_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_erfc_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_exp_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expand_copy_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_expm1_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ifft2_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft2_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfft_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_ihfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft2_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft2_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfft_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fft_rfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fill_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_flip_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fliplr_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_floor_divide_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmax_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmin_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_fmod_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ge_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_geometric_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_gt_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_heaviside_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_imag_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_index_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isclose_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isclose_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isinf_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isinf_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_isnan_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_item_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_item_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lcm_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_le_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_le_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lgamma_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_vecdot_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linalg_vector_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_linspace_tensor_overload_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_log_normal_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logaddexp_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_not_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logical_or_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logspace_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_logsumexp_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_lt_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_masked_fill_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_masked_fill_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_movedim_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_mul_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_narrow_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_native_layer_norm_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_neg_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_empty_strided_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_full_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_new_ones_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_l1_loss_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_layer_norm_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_mish_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pairwise_distance_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pdist_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_pixel_unshuffle_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_smooth_l1_loss_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_nn_functional_softmin_with_dtype_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_ones_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_positive_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_prod_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rad2deg_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rad2deg_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_repeat_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_repeat_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_reshape_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_roll_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rot90_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_round_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_round_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsub_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_rsub_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sgn_cuda_int64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sigmoid_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_signbit_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_signbit_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_signbit_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sin_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sinc_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_entr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_erfcx_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_i1e_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_log_softmax_with_dtype_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_multigammaln_mvlgamma_p_3_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_ndtr_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_special_xlog1py_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_split_with_sizes_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sqrt_cuda_int16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_copy_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_squeeze_copy_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_stack_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_stack_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_sum_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_t_copy_cuda_float16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tan_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_to_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_tril_cuda_int8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_triu_cuda_int32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_true_divide_cuda_uint8, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unflatten_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_copy_cuda_float64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unfold_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_copy_cuda_complex32, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_unsqueeze_copy_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_copy_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_view_cuda_complex128, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_where_cuda_bfloat16, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_zeros_cuda_bool, test/test_ops.py::TestCommonCUDA::test_python_ref_torch_fallback__refs_zeros_cuda_float64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager___rpow___cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_argwhere_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_as_strided_scatter_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_block_diag_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_bool_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cauchy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_constant_pad_nd_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_constant_pad_nd_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_cummin_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_dot_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_empty_strided_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_exp2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_expm1_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_expm1_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_fftshift_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_hfftn_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_fft_ifft_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_ge_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_int_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_jiterator_binary_return_by_ref_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_kthvalue_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_diagonal_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_eig_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_eigvalsh_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_matrix_power_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_norm_subgradients_at_zero_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_qr_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linalg_tensorsolve_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_linspace_tensor_overload_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_log2_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_logspace_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_lu_solve_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_mT_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_sum_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_masked_var_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pad_reflect_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_pad_replicate_negative_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_threshold_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_nn_functional_unfold_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_pca_lowrank_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_permute_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_randn_like_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_ravel_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_rsqrt_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_searchsorted_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sinc_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_sparse_sampled_addmm_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_split_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_transpose_copy_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_trapezoid_cuda_complex64, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_unique_cuda_float32, test/test_ops.py::TestCommonCUDA::test_variant_consistency_eager_vsplit_cuda_complex64, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_einsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_expm1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_ifftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_fft_rfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_frexp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_lu_solve_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_linalg_pinv_singular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_matrix_exp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_minimum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_mul_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_nn_functional_softmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_normal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_qr_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_renorm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_split_with_sizes_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_sum_to_size_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_triu_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_backward_view_as_complex_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_abs_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_diag_embed_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_fft_ihfftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_grid_sampler_2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_isinf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_inv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_lu_factor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_slogdet_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_linalg_solve_triangular_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nanquantile_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_interpolate_bicubic_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_nn_functional_rms_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_randn_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_cow_input_signal_windows_cosine_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_H_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_addcmul_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_all_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_baddbmm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_cummin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_ifft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_fft_ihfft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_index_fill_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_isneginf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_kron_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_eigvals_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_linalg_inv_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_logspace_tensor_overload_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_cumsum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_masked_softmax_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_neg_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_conv2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_nn_functional_hardshrink_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_polygamma_polygamma_n_4_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_round_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_scatter_reduce_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_select_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_signal_windows_gaussian_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_special_chebyshev_polynomial_w_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_std_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_sub_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_t_copy_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_tile_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_to_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_topk_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_trunc_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_forward_ad_unsqueeze_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator__native_batch_norm_legit_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_argmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_double_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_empty_permuted_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_eq_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_fft_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_fft_ifftshift_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_heaviside_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_index_add_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_isreal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_kron_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_lerp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_diagonal_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_eigvals_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_ldl_factor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_svd_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_linalg_vecdot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_logaddexp2_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_adaptive_avg_pool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_pad_constant_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nn_functional_smooth_l1_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_nonzero_static_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_ones_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_permute_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_reshape_as_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_rsub_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_scatter_reduce_prod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_select_scatter_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_sin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_split_with_sizes_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_std_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_svd_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_operator_var_mean_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_H_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_arange_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_conj_physical_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_diag_embed_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_empty_permuted_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fft_fftn_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_fmod_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_isposinf_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_cholesky_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_cross_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_linalg_multi_dot_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_argmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_masked_softmin_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_matmul_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_matrix_exp_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_min_binary_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_mvlgamma_mvlgamma_p_1_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_adaptive_avg_pool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_avg_pool1d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_avg_pool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_group_norm_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_max_unpool2d_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_mish_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_multilabel_margin_loss_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_outer_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_pow_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_rand_like_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_scalar_tensor_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_scatter_reduce_sum_cuda_float32, test/test_ops.py::TestCompositeComplianceCUDA::test_view_replay_special_i1e_cuda_float32, test/test_ops.py::TestMathBitsCUDA::test_conj_view___rdiv___cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs__conversions_byte_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_cat_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_div_no_rounding_mode_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_equal_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_expand_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_fft_ifftshift_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_pow_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view__refs_vstack_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_acosh_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_addcmul_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_as_strided_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cdouble_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_cumprod_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_dot_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_fft_ifft_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_det_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_linalg_householder_product_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_masked_var_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_movedim_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_nansum_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_new_full_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_new_zeros_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_outer_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_permute_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_real_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_resize__cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_resolve_neg_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_sgn_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_square_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_tanh_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_unbind_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_conj_view_var_cuda_complex64, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs__conversions_int_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_any_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_cumprod_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_hsplit_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_linalg_vecdot_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_normal__in_place_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_ones_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_rot90_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_squeeze_multiple_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_std_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_transpose_copy_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_tril_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view__refs_where_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_addbmm_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_addmm_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_cholesky_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fft_fft2_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fft_hfft_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_fill_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_gather_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linalg_vecdot_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_linspace_tensor_overload_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_matmul_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_nn_functional_pad_reflect_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_tril_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_conj_view_triu_cuda_complex128, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_ceil_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_clamp_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_floor_divide_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_log1p_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_log_normal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_maximum_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_narrow_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_channel_shuffle_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_nn_functional_pdist_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_roll_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_special_ndtr_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__refs_tan_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view__segment_reduce_offsets_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_block_diag_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_cat_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_constant_pad_nd_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_contiguous_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_diag_embed_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_equal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_fmod_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_index_select_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_masked_mean_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_adaptive_max_pool2d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_feature_alpha_dropout_with_train_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_max_unpool3d_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_max_unpool3d_grad_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_nn_functional_unfold_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_normal_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_remainder_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_repeat_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_softmax_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_sparse_mm_reduce_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_i1_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_special_shifted_chebyshev_polynomial_t_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_split_with_sizes_copy_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_square_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_svd_lowrank_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_tan_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_transpose_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_trunc_cuda_float64, test/test_ops.py::TestMathBitsCUDA::test_neg_view_var_mean_cuda_float64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast__unsafe_masked_index_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_addbmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_argmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_as_strided_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_atan2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_bitwise_and_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_corrcoef_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_cos_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_ifft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_irfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fft_rfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_fmod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_index_fill_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_isposinf_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_cond_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_linalg_ldl_factor_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_max_pool2d_with_indices_backward_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nextafter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_conv_transpose1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_normal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_permute_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_rand_like_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_softmax_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_special_hermite_polynomial_he_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_var_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_autocast_xlogy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_baddbmm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_bernoulli_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_copysign_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_corrcoef_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_addcdiv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_atan2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_atleast_1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_bfloat16_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_clamp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_clamp_min_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_expand_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_fft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_fft_rfft_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_float_power_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_lu_factor_ex_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_matrix_power_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_linalg_norm_subgradients_at_zero_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_logaddexp2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_masked_cumprod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_matrix_exp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_mvlgamma_mvlgamma_p_5_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_neg_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_adaptive_avg_pool1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_ctc_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_max_unpool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_nn_functional_soft_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_reshape_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_round_decimals_0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_split_with_sizes_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_sub_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_t_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_amp_unsqueeze_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_add_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_as_strided_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_as_strided_partial_views_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_atanh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_div_no_rounding_mode_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_erfinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_fmin_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_frexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_i0_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_index_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_cond_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_pinv_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_qr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_linalg_slogdet_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_log_softmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_sum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_masked_var_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_batch_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_binary_cross_entropy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_channel_shuffle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_dropout2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_fractional_max_pool3d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_kl_div_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_max_unpool1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_max_unpool1d_grad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_nn_functional_silu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_rot90_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_scatter_reduce_prod_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_scatter_reduce_sum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_special_entr_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_tile_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_crossref_backward_no_amp_where_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_div_floor_rounding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_fftshift_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_hfftn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_fft_rfft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_float_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_floor_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_grid_sampler_2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_int_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_item_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_kron_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_eigh_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_log_softmax_with_dtype_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_masked_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_mean_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_meshgrid_list_of_tensors_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_mul_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nan_to_num_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_logsigmoid_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_max_unpool1d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_pad_replicate_negative_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_relu6_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_smooth_l1_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nn_functional_softsign_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_nonzero_static_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_permute_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_polar_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_put_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_resize__cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_round_decimals_3_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_signal_windows_kaiser_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_sparse_mm_reduce_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_bessel_y1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_legendre_polynomial_p_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_shifted_chebyshev_polynomial_u_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_special_zeta_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_svd_lowrank_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_fake_unbind_copy_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_add_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_argmax_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_deg2rad_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_div_floor_rounding_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_dsplit_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_expand_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_fft_ifft2_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_gather_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_heaviside_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_index_add_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_isnan_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_isreal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_linalg_vector_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_logaddexp_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_masked_select_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_matmul_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_maximum_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_batch_norm_without_cudnn_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_feature_alpha_dropout_without_train_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_fractional_max_pool2d_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_glu_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_instance_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_interpolate_area_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_layer_norm_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_pad_replicate_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_pixel_shuffle_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_nn_functional_soft_margin_loss_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_normal_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_outer_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_scatter_add_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_select_scatter_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_bessel_j1_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_laguerre_polynomial_l_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_special_shifted_chebyshev_polynomial_t_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_tensordot_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_trapz_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_pointwise_ops_view_as_cuda_float32, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_arange_cuda_int8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout__refs_logspace_cuda_uint8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_arange_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_full_cuda_int16, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_cuda_int8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_logspace_tensor_overload_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_ones_cuda_int64, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_ones_cuda_uint8, test/test_ops.py::TestFakeTensorCUDA::test_strided_layout_zeros_cuda_int8, test/test_ops.py::TestTagsCUDA::test_tags__refs__conversions_half_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_as_strided_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_dot_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_fft_fft2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_index_select_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_isfinite_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_ne_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_gelu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_nll_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_nn_functional_softmin_with_dtype_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_permute_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_randn_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_signbit_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_i1_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_logit_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_special_zeta_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_t_copy_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags__refs_view_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_baddbmm_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_fft_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_fft_ifft2_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_floor_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_eig_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_ldl_solve_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_lstsq_grad_oriented_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_lu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_pinv_hermitian_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_linalg_tensorsolve_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_long_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_lu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_fill_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_masked_select_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_meshgrid_variadic_tensors_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_min_reduction_with_dim_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_bilinear_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_multilabel_soft_margin_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_nn_functional_poisson_nll_loss_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_polygamma_polygamma_n_1_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_repeat_interleave_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_resize_as__cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_special_airy_ai_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_split_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_sub_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_tensordot_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_triu_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_true_divide_cuda_float32, test/test_ops.py::TestTagsCUDA::test_tags_var_cuda_float32
2025-07-17T09:47:39.4518728Z 
2025-07-17T09:47:40.3683072Z Running test batch 'tests to run' cost 3699.94 seconds
2025-07-17T09:47:40.9876394Z + python test/run_test.py --include inductor/test_torchinductor inductor/test_torchinductor_opinfo inductor/test_aot_inductor --shard 1 2 --verbose
2025-07-17T09:47:43.9793967Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:47:43.9796163Z   import pkg_resources
2025-07-17T09:47:44.6124709Z No TD results found
2025-07-17T09:47:44.6125959Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T09:47:44.6378745Z Found test times from artifacts
2025-07-17T09:47:44.7405433Z Found test times from artifacts
2025-07-17T09:47:44.7444556Z Running all tests
2025-07-17T09:47:44.7450382Z Running parallel tests on 8 processes
2025-07-17T09:47:44.7451436Z Name: tests to run (est. time: 55.35min)
2025-07-17T09:47:44.7451729Z   Serial tests (0):
2025-07-17T09:47:44.7451938Z   Parallel tests (11):
2025-07-17T09:47:44.7452176Z     inductor/test_aot_inductor 1/5
2025-07-17T09:47:44.7452441Z     inductor/test_aot_inductor 3/5
2025-07-17T09:47:44.7452692Z     inductor/test_aot_inductor 4/5
2025-07-17T09:47:44.7455301Z     inductor/test_torchinductor_opinfo 2/17
2025-07-17T09:47:44.7455611Z     inductor/test_torchinductor_opinfo 3/17
2025-07-17T09:47:44.7455895Z     inductor/test_torchinductor_opinfo 6/17
2025-07-17T09:47:44.7456329Z     inductor/test_torchinductor_opinfo 7/17
2025-07-17T09:47:44.7456617Z     inductor/test_torchinductor_opinfo 10/17
2025-07-17T09:47:44.7456910Z     inductor/test_torchinductor_opinfo 11/17
2025-07-17T09:47:44.7457196Z     inductor/test_torchinductor_opinfo 14/17
2025-07-17T09:47:44.7457473Z     inductor/test_torchinductor_opinfo 15/17
2025-07-17T09:47:44.7457748Z Name: excluded (est. time: 0.0min)
2025-07-17T09:47:44.7458001Z   Serial tests (0):
2025-07-17T09:47:44.7458207Z   Parallel tests (0):
2025-07-17T09:47:44.7458528Z Running inductor/test_aot_inductor 1/5 ... [2025-07-17 09:47:44.745312]
2025-07-17T09:47:44.7458910Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:47:44.7459837Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=1', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:47:44.745630]
2025-07-17T09:47:52.2201793Z 
2025-07-17T09:47:52.2203032Z inductor/test_aot_inductor 1/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.5_712d3c69a429b81e_.log
2025-07-17T09:47:52.2204217Z Running 0 items in this shard:
2025-07-17T09:47:52.2204486Z 
2025-07-17T09:47:52.2204859Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T09:47:52.2205548Z Uploading artifacts took 0.00 seconds
2025-07-17T09:47:52.2206176Z Running inductor/test_aot_inductor 3/5 ... [2025-07-17 09:47:52.220443]
2025-07-17T09:47:52.2206823Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:47:52.2210666Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=3', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:47:52.220737]
2025-07-17T09:47:59.5450729Z 
2025-07-17T09:47:59.5451954Z inductor/test_aot_inductor 3/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_3.5_da7dbff49179e661_.log
2025-07-17T09:47:59.5453841Z Running 0 items in this shard:
2025-07-17T09:47:59.5454131Z 
2025-07-17T09:47:59.5463505Z Running inductor/test_aot_inductor 4/5 ... [2025-07-17 09:47:59.545051]
2025-07-17T09:47:59.5464029Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:47:59.5465075Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=4', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:47:59.545343]
2025-07-17T09:48:06.8198344Z 
2025-07-17T09:48:06.8199953Z inductor/test_aot_inductor 4/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_4.5_cf77aced97d4686a_.log
2025-07-17T09:48:06.8201287Z Running 0 items in this shard:
2025-07-17T09:48:06.8201583Z 
2025-07-17T09:48:06.8201986Z Running inductor/test_torchinductor_opinfo 2/17 ... [2025-07-17 09:48:06.819799]
2025-07-17T09:48:06.8202695Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:48:06.8204367Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'serial', '--shard-id=2', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:48:06.820107]
2025-07-17T09:48:16.1972447Z 
2025-07-17T09:48:16.1973789Z inductor/test_torchinductor_opinfo 2/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_2.17_8e2f1cad8887a59f_.log
2025-07-17T09:48:16.1976127Z Running 0 items in this shard:
2025-07-17T09:48:16.1976464Z 
2025-07-17T09:48:16.1976920Z Running inductor/test_torchinductor_opinfo 3/17 ... [2025-07-17 09:48:16.197227]
2025-07-17T09:48:16.1977742Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:48:16.1979711Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'serial', '--shard-id=3', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:48:16.197681]
2025-07-17T09:48:25.5747335Z 
2025-07-17T09:48:25.5748550Z inductor/test_torchinductor_opinfo 3/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_3.17_0ee495d7c740ff30_.log
2025-07-17T09:48:25.5749822Z Running 0 items in this shard:
2025-07-17T09:48:25.5750090Z 
2025-07-17T09:48:25.5750471Z Running inductor/test_torchinductor_opinfo 6/17 ... [2025-07-17 09:48:25.574689]
2025-07-17T09:48:25.5751209Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:48:25.5752853Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'serial', '--shard-id=6', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:48:25.575006]
2025-07-17T09:48:34.9018755Z 
2025-07-17T09:48:34.9020181Z inductor/test_torchinductor_opinfo 6/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_6.17_13e75115af959830_.log
2025-07-17T09:48:34.9021762Z Running 0 items in this shard:
2025-07-17T09:48:34.9022107Z 
2025-07-17T09:48:34.9022568Z Running inductor/test_torchinductor_opinfo 7/17 ... [2025-07-17 09:48:34.901915]
2025-07-17T09:48:34.9023399Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:48:34.9027371Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'serial', '--shard-id=7', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:48:34.902402]
2025-07-17T09:48:45.0805258Z 
2025-07-17T09:48:45.0806502Z inductor/test_torchinductor_opinfo 7/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_7.17_9bfa98cb7a4b9227_.log
2025-07-17T09:48:45.0808364Z Running 0 items in this shard:
2025-07-17T09:48:45.0808656Z 
2025-07-17T09:48:45.0809045Z Running inductor/test_torchinductor_opinfo 10/17 ... [2025-07-17 09:48:45.080530]
2025-07-17T09:48:45.0809753Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:48:45.0811415Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'serial', '--shard-id=10', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:48:45.080855]
2025-07-17T09:48:54.4077188Z 
2025-07-17T09:48:54.4078139Z inductor/test_torchinductor_opinfo 10/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_10.17_722724ba64a8ca03_.log
2025-07-17T09:48:54.4079093Z Running 0 items in this shard:
2025-07-17T09:48:54.4079295Z 
2025-07-17T09:48:54.4079654Z Running inductor/test_torchinductor_opinfo 11/17 ... [2025-07-17 09:48:54.407646]
2025-07-17T09:48:54.4080366Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:48:54.4082211Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'serial', '--shard-id=11', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:48:54.408005]
2025-07-17T09:49:03.7850227Z 
2025-07-17T09:49:03.7851542Z inductor/test_torchinductor_opinfo 11/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_11.17_c67b54684d6b5aeb_.log
2025-07-17T09:49:03.7853416Z Running 0 items in this shard:
2025-07-17T09:49:03.7853704Z 
2025-07-17T09:49:03.7854097Z Running inductor/test_torchinductor_opinfo 14/17 ... [2025-07-17 09:49:03.785012]
2025-07-17T09:49:03.7854819Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:49:03.7856682Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'serial', '--shard-id=14', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:49:03.785317]
2025-07-17T09:49:13.1623721Z 
2025-07-17T09:49:13.1625089Z inductor/test_torchinductor_opinfo 14/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_14.17_b02d3f4659899ac5_.log
2025-07-17T09:49:13.1626632Z Running 0 items in this shard:
2025-07-17T09:49:13.1626968Z 
2025-07-17T09:49:13.1627439Z Running inductor/test_torchinductor_opinfo 15/17 ... [2025-07-17 09:49:13.162300]
2025-07-17T09:49:13.1628276Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:49:13.1630719Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'serial', '--shard-id=15', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:49:13.162817]
2025-07-17T09:49:22.5395324Z 
2025-07-17T09:49:22.5396533Z inductor/test_torchinductor_opinfo 15/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_15.17_a02499edde17ecac_.log
2025-07-17T09:49:22.5397815Z Running 0 items in this shard:
2025-07-17T09:49:22.5398090Z 
2025-07-17T09:49:26.1218634Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:49:26.1220912Z   import pkg_resources
2025-07-17T09:49:26.1662943Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:49:26.1665293Z   import pkg_resources
2025-07-17T09:49:26.3112681Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:49:26.3114697Z   import pkg_resources
2025-07-17T09:49:26.3142890Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:49:26.3145329Z   import pkg_resources
2025-07-17T09:49:26.3147241Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:49:26.3149184Z   import pkg_resources
2025-07-17T09:49:26.3175163Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:49:26.3177568Z   import pkg_resources
2025-07-17T09:49:26.3209602Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:49:26.3211553Z   import pkg_resources
2025-07-17T09:49:26.3377040Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T09:49:26.3379009Z   import pkg_resources
2025-07-17T09:49:26.5368393Z Running inductor/test_aot_inductor 1/5 ... [2025-07-17 09:49:26.536580]
2025-07-17T09:49:26.5368935Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:49:26.5372166Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=1', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:49:26.537005]
2025-07-17T09:49:26.6082445Z Running inductor/test_aot_inductor 3/5 ... [2025-07-17 09:49:26.608090]
2025-07-17T09:49:26.6082925Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:49:26.6086634Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=3', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:49:26.608462]
2025-07-17T09:49:26.8520599Z Running inductor/test_aot_inductor 4/5 ... [2025-07-17 09:49:26.851815]
2025-07-17T09:49:26.8521152Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:49:26.8523977Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=4', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:49:26.852189]
2025-07-17T09:49:26.8835966Z Running inductor/test_torchinductor_opinfo 2/17 ... [2025-07-17 09:49:26.883378]
2025-07-17T09:49:26.8836718Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:49:26.8837401Z Running inductor/test_torchinductor_opinfo 3/17 ... [2025-07-17 09:49:26.883463]
2025-07-17T09:49:26.8838092Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:49:26.8840310Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'not serial', '--shard-id=2', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:49:26.883777]
2025-07-17T09:49:26.8843036Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'not serial', '--shard-id=3', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:49:26.883869]
2025-07-17T09:49:26.8956776Z Running inductor/test_torchinductor_opinfo 6/17 ... [2025-07-17 09:49:26.895504]
2025-07-17T09:49:26.8957453Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:49:26.8961435Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'not serial', '--shard-id=6', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:49:26.895900]
2025-07-17T09:49:26.8963471Z Running inductor/test_torchinductor_opinfo 7/17 ... [2025-07-17 09:49:26.896017]
2025-07-17T09:49:26.8964128Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:49:26.8964853Z Running inductor/test_torchinductor_opinfo 10/17 ... [2025-07-17 09:49:26.896085]
2025-07-17T09:49:26.8965611Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:49:26.8966720Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'not serial', '--shard-id=7', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:49:26.896431]
2025-07-17T09:49:26.8968618Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'not serial', '--shard-id=10', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:49:26.896500]
2025-07-17T09:57:44.1999427Z 
2025-07-17T09:57:44.2000783Z inductor/test_torchinductor_opinfo 3/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_3.17_01683d230bee1fe7_.log
2025-07-17T09:57:44.2094463Z Running 231 items in this shard: test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_H_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___getitem___cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___radd___cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rdiv___cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rdiv___cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmod___cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmul___cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmul___cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rpow___cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rpow___cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rsub___cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rsub___cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__batch_norm_with_update_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__chunk_cat_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_abs_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_abs_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_acos_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addbmm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_alias_copy_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amax_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amin_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_argmin_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_partial_views_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_asinh_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_1d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_1d_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bincount_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bitwise_and_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_block_diag_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_tensors_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_to_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cat_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cat_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_chalf_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_chunk_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_min_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_combinations_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_combinations_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_conj_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_copysign_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_copysign_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_copysign_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_corrcoef_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_corrcoef_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cos_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_count_nonzero_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cross_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cummax_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_deg2rad_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagflat_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_copy_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_scatter_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dot_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_double_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_double_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dsplit_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dstack_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_permuted_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfinv_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expm1_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftshift_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft2_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifftn_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifftn_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifftshift_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifftshift_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfft_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfftn_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfft2_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfft_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfft2_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfft_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfftn_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flipud_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_full_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gather_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ge_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_grid_sampler_2d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_grid_sampler_2d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_half_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_heaviside_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hypot_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_mean_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_select_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_select_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isclose_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isclose_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isnan_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isreal_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_2inputs_2outputs_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_4inputs_with_extra_args_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_binary_return_by_ref_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kron_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ldexp_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ldexp_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_le_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lerp_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_cholesky_ex_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_diagonal_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_inv_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_ldl_factor_ex_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lstsq_grad_oriented_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_vecdot_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linspace_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logcumsumexp_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_xor_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_long_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_long_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lt_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_amin_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_argmax_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_argmax_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumprod_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_fill_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_log_softmax_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_select_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_std_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_std_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_matmul_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_reduction_with_dim_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_binary_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_reduction_with_dim_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_minimum_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mm_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mode_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mvlgamma_mvlgamma_p_3_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nanmedian_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_native_batch_norm_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ne_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_empty_strided_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_full_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_full_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_avg_pool1d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_avg_pool3d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_binary_cross_entropy_with_logits_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv3d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_embedding_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_fractional_max_pool3d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_bilinear_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_linear_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_logsigmoid_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_pool2d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool3d_grad_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_unshuffle_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_selu_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_silu_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softmin_with_dtype_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softplus_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softshrink_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softsign_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_threshold_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_norm_fro_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ones_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_outer_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_prod_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rad2deg_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randint_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ravel_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ravel_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ravel_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_real_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_remainder_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_interleave_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_conj_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_round_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_round_decimals_0_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_short_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sigmoid_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_gaussian_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_hann_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signbit_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sinc_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_slice_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_slice_scatter_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_airy_ai_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_y1_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_w_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_entr_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i0e_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_laguerre_polynomial_l_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_k0_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_polygamma_special_polygamma_n_0_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_scaled_modified_bessel_k0_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_scaled_modified_bessel_k0_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_scaled_modified_bessel_k1_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_v_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_w_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_spherical_bessel_j0_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_zeta_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_square_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_squeeze_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_squeeze_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_squeeze_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_std_mean_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_stft_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_t_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_t_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_take_along_dim_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_take_along_dim_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_take_along_dim_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tensor_split_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tensor_split_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tensor_split_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tile_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_to_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_torch_ops_aten__safe_softmax_default_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapezoid_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapz_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_triangular_solve_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_true_divide_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trunc_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unbind_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unbind_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unfold_copy_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unfold_copy_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsafe_chunk_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsafe_chunk_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_var_mean_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_var_mean_unbiased_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_var_unbiased_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_vdot_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_as_complex_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_vsplit_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_vstack_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zero__cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zeros_cuda_float16
2025-07-17T09:57:44.2181441Z 
2025-07-17T09:57:44.2181666Z Running inductor/test_torchinductor_opinfo 11/17 ... [2025-07-17 09:57:44.200595]
2025-07-17T09:57:44.2182078Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:57:44.2183044Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'not serial', '--shard-id=11', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:57:44.200927]
2025-07-17T09:58:06.7527426Z 
2025-07-17T09:58:06.7528545Z inductor/test_aot_inductor 4/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_4.5_3bc15eaf492deaa6_.log
2025-07-17T09:58:06.7599534Z Running 181 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test__weight_int4pack_mm_with_scales_and_zeros_m_32_n_64_q_group_32_num_groups_1_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_amp_fallback_random_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_aoti_debug_printer_codegen_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_aoti_debug_printer_sym_inputs_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_aoti_debug_printing_model_inputs_codegen_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_aoti_runtime_asserts_backed_symint_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_aoti_runtime_asserts_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_assert_async_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_autotune_with_constant_folding_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_buffer_mutation_and_force_mmap_weights_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_cond_unbacked_symint_closure_dynamic_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_cond_unbacked_symint_closure_dynamic_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_cond_use_buffers_from_outer_scope_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_constant_folding_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_constant_original_fqn_and_dtype_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_constant_type_propagation_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_dynamic_smem_above_default_limit_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_embedding_bag_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_empty_cat_dtype_promotion_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_empty_constant_folding_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_empty_graph_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_fallback_kernel_with_symexpr_output_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_freezing_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_fx_gm_return_tuple_validation_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_input_codegen_with_sympy_expr_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_large_grid_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_nan_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_non_contiguous_output_alias_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_non_default_gpu_device_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_output_path_1_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_repeated_user_defined_triton_kernel_embed_kernel_binary_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_sdpa_2_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_size_with_unbacked_add_expr_transitive_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_small_constant_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_stft_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_sym_i64_input_codegen_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_sympy_cpp_printer_min_max_minmax0_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_equal_to_1_arg_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_1_num_dims_2_dynamic_False_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_1_num_dims_2_dynamic_False_autotune_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_1_num_dims_2_dynamic_True_autotune_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_2_num_dims_2_dynamic_True_autotune_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_3_num_dims_1_dynamic_False_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_3_num_dims_1_dynamic_False_autotune_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_3_num_dims_1_dynamic_True_autotune_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_3_num_dims_2_dynamic_False_autotune_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_3_num_dims_2_dynamic_True_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_3_num_dims_2_dynamic_True_autotune_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_multi_output_arg_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_on_device_tma_dynamic_False_tma_version_new_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_update_user_managed_buffer_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_while_loop_with_outer_buffers_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_while_loop_with_outer_code_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_with_no_triton_profiler_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test__int_mm_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test__weight_int4pack_mm_m_32_n_64_q_group_32_num_groups_1_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_aot_inductor_consts_cpp_build_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_aoti_debug_printer_codegen_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_aoti_profiler_enable_kernel_profile_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_backward_no_op_logging_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_buffer_mutation_and_force_mmap_weights_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_composed_dynamic_size_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_cond_share_predicte_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_cond_with_outer_code_before_after_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_cond_with_parameters_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_deconv_freezing_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_empty_constant_folding_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_fallback_mem_leak_fix_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_fp8_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_fqn_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_int_list_input_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_large_dynamic_dim_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_large_grid_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_linear_freezing_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_load_package_multiple_gpus_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_misc_1_max_autotune_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_misc_1_max_autotune_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_missing_cubin_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_multi_device_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_multiple_output_alias_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_narrow_fallback_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_nested_tensor_from_jagged_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_non_contiguous_output_alias_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_non_default_gpu_device_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_output_misaligned_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_output_path_1_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_profile_benchmark_harness_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_proxy_executor_abs_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_quantized_linear_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_repeat_interleave_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_repeat_output_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_return_constant_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_runtime_checks_fp8_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_runtime_checks_large_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_same_backing_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_seq_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_simple_multi_arch_embed_kernel_binary_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_symint_item_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_sympy_cpp_printer_min_max_minmax0_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_sympy_cpp_printer_min_max_minmax1_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_dynamic_grid_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_equal_to_1_float_arg_dynamic_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_equal_to_1_float_arg_dynamic_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_1_num_dims_2_dynamic_False_autotune_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_2_num_dims_1_dynamic_False_autotune_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_on_device_tma_dynamic_False_tma_version_new_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_on_device_tma_dynamic_True_tma_version_new_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_reinterpret_view_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_tma_descriptor_2d_dynamic_True_tma_version_new_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_unbacked_symint_in_grid_dynamic_False_autotuning_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_unbacked_symint_in_grid_dynamic_False_autotuning_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_while_loop_with_outer_buffers_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_while_loop_with_pytree_inputs_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_while_loop_with_sym_expr_cond_dynamic_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_with_no_triton_profiler_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_addmm_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_aoti_debug_printer_codegen_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_aoti_debug_printer_fp8_dtype_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_aoti_debug_printer_sym_inputs_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_autotuning_args_reuse_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_bool_input_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_boolean_indexing_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_buffer_mutation_3_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_composed_dynamic_size_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_cond_share_predicte_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_cond_use_buffers_from_outer_scope_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_cond_with_multiple_outputs_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_cond_with_reinterpret_view_inputs_outputs_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_constant_folding_with_update_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_conv3d_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_conv_freezing_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_deconv_freezing_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_dynamic_scalar_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_dynamic_smem_above_default_limit_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_empty_cat_dtype_promotion_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_fp8_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_fp8_view_of_param_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_index_put_fallback_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_input_codegen_with_sympy_expr_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_linear_dynamic_maxautotune_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_misc_1_max_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_missing_output_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_model_modified_weights_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_multi_device_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_narrow_fallback_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_nested_tensor_from_jagged_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_quanatized_int8_linear_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_return_view_constant_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_reuse_kernel_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_runtime_checks_fp8_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_runtime_checks_large_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_runtime_checks_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_simple_embed_kernel_binary_True_max_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_simple_split_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_size_with_unbacked_add_expr_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_size_with_unbacked_add_expr_transitive_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_stride_with_unbacked_expr_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_sym_i64_input_codegen_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_symfloat_item_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_dynamic_grid_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_dynamic_shape_with_div_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_equal_to_1_float_arg_dynamic_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_extern_kernel_arg_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_1_num_dims_1_dynamic_False_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_3_num_dims_1_dynamic_True_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_3_num_dims_2_dynamic_True_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_on_device_tma_dynamic_True_tma_version_old_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_reinterpret_view_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_sympy_expr_arg_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_tma_descriptor_1d_dynamic_True_tma_version_new_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_tma_descriptor_1d_dynamic_True_tma_version_old_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_update_inactive_constant_buffer_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_view_outputs_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_weight_on_disk_legacy_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_while_loop_nested_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_while_loop_simple_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_while_loop_with_mixed_device_dynamic_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_while_loop_with_mixed_device_dynamic_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_while_loop_with_parameters_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_with_no_triton_profiler_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_zero_size_weight_mps
2025-07-17T09:58:06.7665149Z 
2025-07-17T09:58:06.7665378Z Running inductor/test_torchinductor_opinfo 14/17 ... [2025-07-17 09:58:06.753237]
2025-07-17T09:58:06.7665788Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:58:06.7666745Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'not serial', '--shard-id=14', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:58:06.753647]
2025-07-17T09:58:23.2625573Z 
2025-07-17T09:58:23.2626825Z inductor/test_aot_inductor 1/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.5_d0a1d946909072ab_.log
2025-07-17T09:58:23.2704309Z Running 161 items in this shard: test/inductor/test_aot_inductor.py::TestAOTInductorConfig::test_compile_standalone_package_cpp_already_true, test/inductor/test_aot_inductor.py::TestAOTInductorConfig::test_no_compile_standalone, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test__weight_int4pack_mm_with_scales_and_zeros_m_32_n_64_q_group_64_num_groups_1_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test__weight_int4pack_mm_with_scales_and_zeros_m_32_n_64_q_group_64_num_groups_2_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_aot_inductor_consts_cpp_build_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_bmm_multiple_dynamic_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_buffer_reuse_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_cond_mismatched_branch_output_dynamic_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_cond_nested_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_cond_with_outer_code_before_after_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_conv3d_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_dup_unbacked_sym_decl_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_duplicate_constant_folding_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_dynamic_cat_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_extract_constants_map_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_fallback_mem_leak_fix_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_fill__fallback_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_free_inactive_buffer_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_index_put_fallback_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_issue_140766_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_large_weight_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_load_package_multiple_gpus_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_masked_select_dynamic_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_misc_1_max_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_quanatized_int8_linear_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_quantized_linear_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_repeated_calling_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_reuse_kernel_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_runtime_checks_complex_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_same_backing_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_sdpa_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_simple_embed_kernel_binary_True_max_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_simple_embed_kernel_binary_True_max_autotune_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_simple_multi_arch_embed_kernel_binary_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_simple_split_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_size_with_unbacked_add_expr_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_subclasses_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_autotuning_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_equal_to_1_float_arg_dynamic_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_1_num_dims_1_dynamic_False_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_1_num_dims_1_dynamic_True_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_1_num_dims_1_dynamic_True_autotune_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_2_num_dims_1_dynamic_False_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_3_num_dims_1_dynamic_True_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_3_num_dims_2_dynamic_False_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_on_device_tma_dynamic_False_tma_version_old_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_on_device_tma_dynamic_True_tma_version_new_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_reinterpret_view_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_sympy_expr_arg_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_tma_descriptor_1d_dynamic_False_tma_version_old_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_tma_descriptor_2d_dynamic_False_tma_version_old_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_unbacked_symint_in_grid_dynamic_False_autotuning_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_unbacked_symint_in_grid_dynamic_True_autotuning_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_unbacked_symint_in_grid_dynamic_True_autotuning_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_weird_param_order_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_next_power_of_2_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_with_profiler_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_zero_size_weight_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_addmm_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_aoti_debug_printing_model_inputs_codegen_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_aoti_runtime_asserts_backed_symint_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_autotuning_args_reuse_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_boolean_indexing_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_buffer_mutation_3_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_cond_non_tensor_predicates_dynamic_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_cond_unbacked_symint_closure_dynamic_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_consecutive_compiles_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_constant_folding_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_convolution_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_dup_unbacked_sym_decl_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_duplicated_params_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_extract_constants_map_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_index_put_fallback_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_masked_select_dynamic_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_misaligned_input_1_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_no_args_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_normal_functional_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_proxy_executor_hann_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_proxy_executor_permute_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_repeated_user_defined_triton_kernel_embed_kernel_binary_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_reuse_kernel_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_reuse_kernel_dynamic_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_run_with_grad_enabled_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_simple_embed_kernel_binary_True_max_autotune_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_size_from_multi_output_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_size_with_unbacked_add_expr_transitive_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_small_constant_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_stride_with_unbacked_expr_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_symbool_item_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_symfloat_item_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_autotuning_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_1_num_dims_1_dynamic_False_autotune_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_1_num_dims_2_dynamic_False_autotune_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_2_num_dims_2_dynamic_True_autotune_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_3_num_dims_1_dynamic_False_autotune_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_3_num_dims_1_dynamic_True_autotune_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_3_num_dims_2_dynamic_False_autotune_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_on_device_tma_dynamic_False_tma_version_old_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_tma_descriptor_2d_dynamic_False_tma_version_new_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_tma_descriptor_2d_dynamic_True_tma_version_old_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_unbacked_symint_in_grid_dynamic_True_autotuning_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_mutated_autotuning_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_next_power_of_2_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_update_inactive_constant_buffer_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_while_loop_simple_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_while_loop_with_conv_dynamic_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_while_loop_with_mixed_device_dynamic_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_while_loop_with_unbacked_symint_closure_dynamic_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_with_offset_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_zero_grid_with_backed_symbols_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_zero_grid_with_unbacked_symbols_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test__int_mm_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test__weight_int4pack_mm_m_32_n_64_q_group_32_num_groups_1_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test__weight_int4pack_mm_m_32_n_64_q_group_32_num_groups_2_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test__weight_int4pack_mm_m_32_n_64_q_group_64_num_groups_1_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_addmm_multiple_dynamic_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_aot_inductor_consts_cpp_build_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_aoti_debug_printing_model_inputs_codegen_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_aoti_profiler_enable_kernel_profile_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_assert_tensor_meta_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_bmm_multiple_dynamic_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_buffer_mutation_1_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_clamp_decomposition_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_cond_unbacked_symint_closure_dynamic_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_cond_with_parameters_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_constant_folding_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_constant_original_fqn_and_dtype_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_duplicate_constant_folding_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_embedding_bag_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_empty_graph_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_fallback_mem_leak_fix_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_freezing_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_int_list_input_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_misc_1_max_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_nan_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_output_path_1_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_poi_multiple_dynamic_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_profile_benchmark_harness_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_proxy_executor_squeeze_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_pytree_inputs_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_quantized_linear_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_repeated_calling_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_seq_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_simple_dynamic_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_simple_embed_kernel_binary_False_max_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_stft_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_sympy_cpp_printer_min_max_minmax1_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_2_num_dims_1_dynamic_False_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_2_num_dims_2_dynamic_False_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_2_num_dims_2_dynamic_False_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_2_num_dims_2_dynamic_True_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_3_num_dims_1_dynamic_False_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_3_num_dims_1_dynamic_True_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_3_num_dims_2_dynamic_False_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_on_device_tma_dynamic_False_tma_version_new_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_unbacked_symint_in_grid_dynamic_True_autotuning_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_unbacked_symint_in_grid_dynamic_True_autotuning_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_while_loop_with_conv_dynamic_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_while_loop_with_outer_code_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_with_offset_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_zero_size_buffer_mps
2025-07-17T09:58:23.2763521Z 
2025-07-17T09:58:23.2763746Z Running inductor/test_torchinductor_opinfo 15/17 ... [2025-07-17 09:58:23.263017]
2025-07-17T09:58:23.2764171Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T09:58:23.2765144Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor_opinfo.py', '-m', 'not serial', '--shard-id=15', '--num-shards=17', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 09:58:23.263334]
2025-07-17T09:59:11.2141269Z 
2025-07-17T09:59:11.2142923Z inductor/test_torchinductor_opinfo 2/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_2.17_9b14c93939321457_.log
2025-07-17T09:59:11.2228239Z Running 196 items in this shard: test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_H_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_H_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___ror___cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__unsafe_masked_index_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__unsafe_masked_index_put_accumulate_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_acos_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_angle_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_arange_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_argwhere_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_asinh_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan2_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan2_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_block_diag_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bmm_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_tensors_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cartesian_prod_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cdist_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ceil_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_char_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_chunk_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_min_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clone_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_column_stack_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_conj_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_contiguous_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_corrcoef_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cos_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_count_nonzero_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cov_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cov_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cummax_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cumprod_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diag_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diag_embed_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagflat_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_div_floor_rounding_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_div_trunc_rounding_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_div_trunc_rounding_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dstack_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_like_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_strided_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_eq_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erf_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erf_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_exp_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fft2_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftn_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfft2_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfft_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft2_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifftn_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfft2_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfft_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfft_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flip_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmax_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_full_like_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gather_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gradient_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gradient_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_histc_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hsplit_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hsplit_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hstack_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hstack_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_i0_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_add_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_prod_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_inner_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isfinite_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isin_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isnan_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ldexp_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lgamma_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_diagonal_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_eigvalsh_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lstsq_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lu_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lu_factor_ex_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_matrix_power_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_slogdet_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_vander_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log10_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log_normal_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_not_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_or_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logit_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logspace_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logsumexp_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lt_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lu_solve_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_amax_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumprod_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumprod_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumprod_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_scatter_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_sum_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_matmul_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_matrix_exp_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mode_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_movedim_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nanmean_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_ones_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_zeros_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_zeros_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_zeros_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nextafter_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_batch_norm_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_celu_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_cosine_similarity_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_ctc_loss_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_feature_alpha_dropout_without_train_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_feature_alpha_dropout_without_train_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_grid_sample_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_nearest_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_l1_loss_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_leaky_relu_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_margin_ranking_loss_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_margin_ranking_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_pool3d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool3d_grad_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_unshuffle_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_relu6_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_relu_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_relu_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_rms_norm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_scaled_dot_product_attention_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_upsample_nearest_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_upsample_nearest_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ones_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_2_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_4_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_qr_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randint_like_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randn_like_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_remainder_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_renorm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_reshape_as_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_neg_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_round_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_round_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_round_decimals_neg_3_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_amin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_sum_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_searchsorted_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_scatter_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_blackman_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_hamming_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sinc_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sinh_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_slice_scatter_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sort_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_y1_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_y1_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_y1_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_t_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_v_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_v_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_v_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_v_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_erfcx_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_i0_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_i1_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_k0_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_k0_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_k0_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_k1_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_scaled_modified_bessel_k0_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_w_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_xlog1py_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_with_sizes_copy_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_std_unbiased_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sum_to_size_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_svd_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tanh_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tensordot_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_torch_ops_aten__safe_softmax_default_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trace_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_triu_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_true_divide_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unbind_copy_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unbind_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unflatten_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unflatten_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unfold_copy_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_consecutive_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsafe_chunk_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsqueeze_copy_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_copy_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zero__cuda_bool
2025-07-17T09:59:11.2302503Z 
2025-07-17T09:59:12.5539997Z 
2025-07-17T09:59:12.5541006Z inductor/test_aot_inductor 3/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_3.5_6b9d44f1f7078b9a_.log
2025-07-17T09:59:12.5605330Z Running 159 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorLoggingTest::test_shape_env_reuse, test/inductor/test_aot_inductor.py::TestAOTInductorConfig::test_compile_standalone_package_cpp_false_raises, test/inductor/test_aot_inductor.py::TestAOTInductorConfig::test_compile_standalone_sets_package_cpp, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test__weight_int4pack_mm_m_32_n_64_q_group_32_num_groups_2_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_add_complex_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_addmm_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_aoti_constant_tensor_name_collision_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_bool_input_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_buffer_mutation_2_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_clamp_decomposition_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_cond_non_tensor_predicates_dynamic_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_cond_with_parameters_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_cond_with_reinterpret_view_inputs_outputs_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_consecutive_compiles_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_constant_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_constant_folding_with_update_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_conv_freezing_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_fft_c2c_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_fp8_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_fp8_view_of_param_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_misaligned_input_1_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_misc_1_max_autotune_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_model_modified_weights_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_multi_device_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_normal_functional_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_output_misaligned_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_proxy_executor_hann_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_pytree_inputs_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_run_with_grad_enabled_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_runtime_checks_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_runtime_checks_device_type_failed_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_runtime_checks_dtype_failed_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_runtime_checks_fp8_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_scatter_reduce_fallback_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_simple_dynamic_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_stride_with_unbacked_expr_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_symbool_item_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_symint_item_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_dynamic_launcher_grid_infer_from_tensor_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_dynamic_grid_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_dynamic_shape_with_div_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_1_num_dims_2_dynamic_True_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_grid_type_2_num_dims_1_dynamic_True_autotune_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_reinterpret_view_mem_leak_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_tma_descriptor_2d_dynamic_False_tma_version_new_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_tma_descriptor_2d_dynamic_True_tma_version_old_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_kernel_unbacked_symint_in_grid_dynamic_False_autotuning_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_triton_mutated_autotuning_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_update_constant_buffer_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_using_model_name_for_files_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_view_outputs_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_while_loop_with_conv_dynamic_True_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_while_loop_with_mixed_device_dynamic_False_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_while_loop_with_parameters_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_while_loop_with_pytree_inputs_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test__weight_int4pack_mm_with_scales_and_zeros_m_32_n_64_q_group_32_num_groups_1_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test__weight_int4pack_mm_with_scales_and_zeros_m_32_n_64_q_group_32_num_groups_2_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_add_complex_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_addmm_multiple_dynamic_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_aoti_constant_tensor_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_aoti_profiler_enable_kernel_profile_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_autotune_with_constant_folding_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_buffer_mutation_4_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_clamp_decomposition_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_cond_mismatched_branch_output_dynamic_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_cond_nested_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_cond_use_buffers_from_outer_scope_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_cond_with_multiple_outputs_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_constant_folding_with_update_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_constant_original_fqn_and_dtype_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_conv3d_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_duplicate_constant_folding_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_dynamic_cat_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_dynamic_smem_above_default_limit_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_empty_graph_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_foreach_multiple_dynamic_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_free_inactive_buffer_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_linear_dynamic_maxautotune_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_missing_output_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_model_modified_weights_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_output_path_2_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_proxy_executor_squeeze_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_repeated_calling_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_repeated_user_defined_triton_kernel_embed_kernel_binary_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_return_view_constant_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_runtime_checks_dtype_failed_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_runtime_checks_shape_failed_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_scatter_reduce_fallback_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_sdpa_2_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_sdpa_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_simple_dynamic_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_simple_embed_kernel_binary_False_max_autotune_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_stft_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_subclasses_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_sym_i64_input_codegen_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_extern_kernel_arg_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_1_num_dims_1_dynamic_True_autotune_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_3_num_dims_1_dynamic_True_autotune_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_grid_type_3_num_dims_2_dynamic_True_autotune_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_on_device_tma_dynamic_True_tma_version_old_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_reinterpret_view_mem_leak_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_tma_descriptor_1d_dynamic_True_tma_version_new_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_tma_descriptor_1d_dynamic_True_tma_version_old_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_triton_kernel_unbacked_symint_in_grid_dynamic_True_autotuning_False_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_while_loop_with_outer_code_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_while_loop_with_unbacked_symint_closure_dynamic_True_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_with_profiler_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test__weight_int4pack_mm_m_32_n_64_q_group_64_num_groups_2_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test__weight_int4pack_mm_with_scales_and_zeros_m_32_n_64_q_group_32_num_groups_2_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_add_complex_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_aoti_debug_printer_cpp_kernel_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_aoti_runtime_asserts_backed_symint_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_buffer_mutation_4_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_cond_non_tensor_predicates_dynamic_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_consecutive_compiles_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_constant_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_constant_type_propagation_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_dup_unbacked_sym_decl_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_duplicated_params_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_empty_constant_folding_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_extract_constants_map_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_fill__fallback_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_fx_gm_return_tuple_validation_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_inf_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_large_grid_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_linear_freezing_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_load_package_multiple_gpus_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_non_contiguous_output_alias_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_none_args_aot_codegen_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_normal_functional_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_output_misaligned_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_pad_fallback_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_proxy_executor_abs_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_run_with_grad_enabled_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_runtime_checks_complex_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_scaled_dot_product_efficient_attention_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_scatter_reduce_fallback_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_sdpa_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_simple_multi_arch_embed_kernel_binary_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_size_with_unbacked_add_and_mul_expr_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_symint_item_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_autotuning_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_dynamic_launcher_grid_infer_from_tensor_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_dynamic_launcher_grid_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_equal_to_1_arg_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_1_num_dims_1_dynamic_False_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_1_num_dims_1_dynamic_True_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_1_num_dims_2_dynamic_True_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_1_num_dims_2_dynamic_True_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_2_num_dims_1_dynamic_False_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_2_num_dims_1_dynamic_True_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_3_num_dims_1_dynamic_False_autotune_True_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_grid_type_3_num_dims_2_dynamic_True_autotune_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_multi_output_arg_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_kernel_unbacked_symint_in_grid_dynamic_False_autotuning_False_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_triton_mutated_autotuning_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_while_loop_with_pytree_inputs_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_with_profiler_mps, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_zero_grid_with_backed_symbols_mps
2025-07-17T09:59:12.5663115Z 
2025-07-17T10:00:21.9891993Z 
2025-07-17T10:00:21.9893420Z inductor/test_torchinductor_opinfo 7/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_7.17_b46fb9bd561901b6_.log
2025-07-17T10:00:21.9978494Z Running 209 items in this shard: test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___radd___cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rand___cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmod___cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmul___cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rxor___cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__batch_norm_with_update_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__batch_norm_with_update_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__segment_reduce_lengths_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__unsafe_masked_index_put_accumulate_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addcmul_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addr_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_all_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amax_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_angle_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_argsort_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_argwhere_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_copy_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan2_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_1d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_3d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_3d_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bernoulli_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bitwise_and_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bitwise_left_shift_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bitwise_right_shift_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bitwise_xor_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_block_diag_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bucketize_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bucketize_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cfloat_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_max_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_min_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_column_stack_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cummax_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cummin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cumulative_trapezoid_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_deg2rad_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diag_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_digamma_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dot_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dsplit_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dsplit_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_like_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_permuted_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_strided_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_strided_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erf_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_as_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_eye_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fft2_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftshift_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfft2_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfftn_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfftn_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfft_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfft_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfftn_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fill_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmin_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmod_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ge_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_add_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_fill_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_amax_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_mean_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_select_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_select_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isclose_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isfinite_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isinf_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isposinf_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isreal_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isreal_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_4inputs_with_extra_args_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_binary_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kthvalue_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ldexp_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_diagonal_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_matrix_norm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_matrix_rank_hermitian_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_norm_subgradients_at_zero_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linspace_tensor_overload_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log10_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logdet_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_not_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_not_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_or_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logspace_tensor_overload_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_long_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_amin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumsum_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumsum_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumsum_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_normalize_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_prod_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_sum_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_var_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_binary_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_pool2d_with_indices_backward_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_reduction_no_dim_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_maximum_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_maximum_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_meshgrid_list_of_tensors_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_movedim_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mvlgamma_mvlgamma_p_5_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nanmedian_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nanmedian_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_empty_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_max_pool1d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_avg_pool1d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_avg_pool1d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_avg_pool3d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_binary_cross_entropy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_channel_shuffle_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_channel_shuffle_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv1d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv2d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv_transpose3d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_gaussian_nll_loss_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_hardsigmoid_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_huber_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_bilinear_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_nearest-exact_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_trilinear_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_l1_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_linear_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_local_response_norm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_pool2d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool1d_grad_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool1d_grad_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_one_hot_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_constant_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_reflect_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pdist_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_poisson_nll_loss_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_poisson_nll_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_poisson_nll_loss_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_prelu_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_smooth_l1_loss_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_smooth_l1_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softplus_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_triplet_margin_loss_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ones_like_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_permute_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_3_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randint_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randint_like_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_reciprocal_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_reciprocal_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_interleave_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resize_as__cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_conj_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_roll_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rot90_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rsqrt_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scalar_tensor_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_add_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_amin_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_mean_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_searchsorted_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sgn_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sgn_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_short_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_blackman_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_general_hamming_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_hann_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sort_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_j0_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_u_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_erfcx_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_hermite_polynomial_h_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i1_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i1_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_i1_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_scaled_modified_bessel_k0_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_zeta_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_list_args_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_square_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_square_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_squeeze_multiple_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_stack_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_stack_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sub_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_t_copy_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_take_along_dim_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_take_along_dim_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_take_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tanh_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tensor_split_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tile_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_to_sparse_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_torch_ops_aten__safe_softmax_default_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trace_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapz_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_true_divide_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unfold_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_consecutive_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_cuda_uint16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsqueeze_copy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_var_mean_unbiased_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_xlogy_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_xlogy_cuda_float32
2025-07-17T10:00:22.0057983Z 
2025-07-17T10:01:25.0649131Z 
2025-07-17T10:01:25.0650292Z inductor/test_torchinductor_opinfo 6/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_6.17_360b03bfdd160bf5_.log
2025-07-17T10:01:25.0739940Z Running 211 items in this shard: test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_T_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rand___cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmul___cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmul___cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rpow___cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rsub___cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__chunk_cat_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__softmax_backward_data_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addcdiv_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addmv_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addr_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_allclose_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amin_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_any_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_any_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atanh_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atanh_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_2d_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bitwise_and_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bitwise_not_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bitwise_not_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_shapes_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cat_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_chalf_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_char_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cholesky_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cholesky_solve_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_max_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_column_stack_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_combinations_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_copysign_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_copysign_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cov_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cummin_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cumsum_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cumulative_trapezoid_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_deg2rad_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diag_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_scatter_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_div_no_rounding_mode_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_double_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dstack_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_eq_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erf_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfc_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfinv_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_exp2_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_as_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_as_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_eye_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fft2_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfft2_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfft_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfft_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfft_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfft_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfftn_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fill_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fill_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flatten_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flip_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fliplr_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_power_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmax_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmod_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmod_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmod_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_full_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_full_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_geometric_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_geqrf_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hsplit_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_copy_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_put_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_amin_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isnan_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isnan_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isposinf_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_2inputs_2outputs_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_2inputs_2outputs_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_2inputs_2outputs_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_4inputs_with_extra_args_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_binary_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_binary_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lstsq_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_vander_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_vector_norm_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linspace_tensor_overload_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linspace_tensor_overload_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log1p_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log1p_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_and_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_and_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_not_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logit_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logit_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logit_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logspace_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logspace_tensor_overload_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lu_unpack_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_argmin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_logaddexp_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_logsumexp_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_prod_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_std_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_median_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_reduction_no_dim_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_reduction_no_dim_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_minimum_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_minimum_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_movedim_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_multinomial_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mv_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mv_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mvlgamma_mvlgamma_p_3_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nansum_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_copy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ne_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_empty_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_full_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_zeros_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_avg_pool1d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_max_pool1d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_max_pool2d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_avg_pool3d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_channel_shuffle_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv2d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv_transpose1d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv_transpose2d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_dropout_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_glu_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_hardshrink_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_hardtanh_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_margin_ranking_loss_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool1d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool2d_grad_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool3d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_multi_margin_loss_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_multilabel_soft_margin_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_circular_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_reflect_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_replicate_negative_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pairwise_distance_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pairwise_distance_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_relu6_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_rrelu_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_silu_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softmin_with_dtype_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softplus_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_tanhshrink_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_triplet_margin_with_distance_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_static_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_norm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_normal_number_mean_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_2_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_2_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_2_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_4_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randint_like_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_interleave_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resize__cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resize__cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_conj_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rsqrt_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_add_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_amin_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_mean_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_prod_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sgn_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_short_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sigmoid_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sin_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sinc_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sort_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sort_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sort_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_j0_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_hermite_polynomial_h_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i1e_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_k1_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_scaled_modified_bessel_k1_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_scaled_modified_bessel_k1_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_t_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_zeta_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_with_sizes_copy_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_svd_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_t_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_t_copy_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_t_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tensordot_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_to_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_transpose_copy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_transpose_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_transpose_copy_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_transpose_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsafe_split_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsqueeze_copy_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_var_unbiased_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zeros_cuda_int32
2025-07-17T10:01:25.0819646Z 
2025-07-17T10:01:31.0957298Z 
2025-07-17T10:01:31.0958662Z inductor/test_torchinductor_opinfo 10/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_10.17_9d9acaf17cf5399a_.log
2025-07-17T10:01:31.1058323Z Running 225 items in this shard: test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rdiv___cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rxor___cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__native_batch_norm_legit_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_acos_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_acosh_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addmm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_all_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_allclose_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amin_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_angle_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_arange_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_arange_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_partial_views_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_scatter_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_scatter_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan2_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bfloat16_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bfloat16_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_block_diag_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bool_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bool_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_tensors_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_to_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bucketize_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_chalf_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_chalf_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_min_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_combinations_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_combinations_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_conj_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_conj_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_conj_physical_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_conj_physical_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_constant_pad_nd_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_copysign_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_copysign_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_corrcoef_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_count_nonzero_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diag_embed_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagflat_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagflat_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diff_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diff_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dsplit_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_permuted_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_strided_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_strided_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_equal_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfc_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfinv_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfinv_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfinv_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expm1_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftn_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftn_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftn_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftshift_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfftn_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfft2_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfftn_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfftn_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfft2_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flatten_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flatten_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flip_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_power_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_power_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmod_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gather_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gcd_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ge_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_geometric_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gt_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_histc_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hypot_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_i0_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_i0_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_igamma_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_put_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_put_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_amin_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_mean_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isfinite_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isinf_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isreal_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_binary_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_binary_return_by_ref_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kron_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_le_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_cholesky_ex_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_inv_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lu_factor_ex_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_matrix_rank_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_multi_dot_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_norm_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_norm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_pinv_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_qr_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_solve_ex_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_svdvals_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_vector_norm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log_normal_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log_softmax_with_dtype_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logspace_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logspace_tensor_overload_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lt_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mH_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mH_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_argmax_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_argmin_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumsum_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_fill_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_logsumexp_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_prod_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_matrix_exp_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_reduction_with_dim_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_reduction_with_dim_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_median_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_meshgrid_list_of_tensors_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_meshgrid_variadic_tensors_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_movedim_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_msort_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mvlgamma_mvlgamma_p_1_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mvlgamma_mvlgamma_p_5_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_copy_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ne_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_full_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_max_pool3d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_cosine_embedding_loss_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_dropout2d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_embedding_bag_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_feature_alpha_dropout_with_train_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_hinge_embedding_loss_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_leaky_relu_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_leaky_relu_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_pool1d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool3d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_circular_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_shuffle_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_shuffle_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_relu_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_tanhshrink_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_tanhshrink_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_upsample_nearest_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_0_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_4_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_pow_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_pow_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_prod_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_put_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rad2deg_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rad2deg_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randn_like_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_reciprocal_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_reshape_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resize_as__cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resize_as__cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_neg_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rot90_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_round_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rsub_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rsub_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rsub_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_prod_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_sum_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_scatter_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_scatter_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_scatter_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_cosine_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signbit_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sin_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_slice_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_slice_scatter_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_slice_scatter_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_softmax_with_dtype_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_j1_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_hermite_polynomial_he_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i1e_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_laguerre_polynomial_l_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_k1_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_polygamma_special_polygamma_n_0_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_v_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_xlog1py_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_list_args_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_with_sizes_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_with_sizes_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sqrt_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_squeeze_copy_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_squeeze_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_std_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_std_unbiased_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sum_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sum_to_size_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sum_to_size_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_to_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapezoid_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tril_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tril_indices_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_triu_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trunc_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unfold_copy_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_consecutive_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsqueeze_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_var_mean_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_as_complex_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_vsplit_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_where_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_xlogy_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zeros_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zeros_like_cuda_float16
2025-07-17T10:01:31.1142168Z 
2025-07-17T10:05:18.0613170Z 
2025-07-17T10:05:18.0615047Z inductor/test_torchinductor_opinfo 14/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_14.17_30380a4869273394_.log
2025-07-17T10:05:18.0700029Z Running 205 items in this shard: test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___getitem___cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmod___cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rxor___cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__segment_reduce_offsets_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__softmax_backward_data_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_add_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addcmul_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addcmul_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_aminmax_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_any_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atanh_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_1d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_1d_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_2d_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_3d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bincount_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_tensors_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bucketize_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_byte_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cartesian_prod_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cartesian_prod_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cauchy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_char_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_char_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_min_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_column_stack_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cummax_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cumsum_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_deg2rad_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_deg2rad_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diag_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diag_embed_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diag_embed_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_scatter_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_div_trunc_rounding_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_strided_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_equal_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_erfc_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_exponential_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fft2_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftn_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfft2_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfftn_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfftn_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifftshift_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfft2_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfft_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfft_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfftn_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfft2_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_rfftn_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fill_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flatten_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fliplr_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fliplr_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_power_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_floor_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmod_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gather_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gradient_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_half_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_heaviside_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_histc_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_histc_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hstack_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_i0_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_amin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isclose_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isin_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isnan_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isposinf_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isposinf_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_item_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_4inputs_with_extra_args_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_binary_return_by_ref_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_binary_return_by_ref_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_unary_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kron_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_cross_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_householder_product_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lu_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lu_factor_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lu_solve_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_svdvals_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linspace_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log2_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log2_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log2_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log_softmax_with_dtype_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_xor_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logspace_tensor_overload_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logsumexp_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logsumexp_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_fill_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_log_softmax_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_logsumexp_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_median_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_select_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_select_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_var_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_matrix_exp_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_binary_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_reduction_with_dim_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_maximum_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mean_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_meshgrid_list_of_tensors_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_meshgrid_list_of_tensors_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_binary_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_reduction_no_dim_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_multinomial_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mv_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_empty_strided_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_ones_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_avg_pool1d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_avg_pool2d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_max_pool2d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_binary_cross_entropy_with_logits_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_channel_shuffle_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv_transpose2d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv_transpose3d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_cosine_embedding_loss_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_cosine_similarity_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_dropout3d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_dropout_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_elu_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_feature_alpha_dropout_without_train_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_hardtanh_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_bicubic_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_l1_loss_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_margin_ranking_loss_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_pool3d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_normalize_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_circular_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_constant_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_reflect_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_replicate_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pairwise_distance_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_shuffle_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_relu6_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_triplet_margin_with_distance_loss_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_static_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_normal_number_mean_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ones_like_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_pinverse_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_1_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_3_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_put_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_put_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_remainder_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_renorm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_interleave_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_reshape_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resize_as__cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_neg_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_sum_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sigmoid_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sign_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sign_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_general_hamming_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_signal_windows_nuttall_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sin_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_softmax_with_dtype_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sparse_sampled_addmm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_t_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_u_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_erfcx_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i0e_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i1_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_log_ndtr_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_i0_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_i1_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_ndtr_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_ndtr_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_ndtri_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_with_sizes_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_with_sizes_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sqrt_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_square_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sub_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_take_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tanh_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tile_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_topk_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_torch_ops_aten__efficient_attention_forward_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_torch_ops_aten__flash_attention_forward_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_torch_ops_aten__safe_softmax_default_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trace_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_transpose_copy_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unbind_copy_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unfold_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_uniform_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsafe_split_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsqueeze_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsqueeze_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_as_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_xlogy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zeros_cuda_uint8
2025-07-17T10:05:18.0777810Z 
2025-07-17T10:06:48.2618467Z 
2025-07-17T10:06:48.2619870Z inductor/test_torchinductor_opinfo 15/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_15.17_1389a86e1c4bd188_.log
2025-07-17T10:06:48.2707438Z Running 209 items in this shard: test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_H_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___radd___cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rand___cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rdiv___cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rsub___cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__chunk_cat_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__segment_reduce_lengths_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__unsafe_masked_index_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__upsample_bilinear2d_aa_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_abs_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_acosh_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addcmul_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_alias_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_alias_copy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_all_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_all_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_all_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amax_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_aminmax_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_arange_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_arange_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_argsort_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_copy_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_asin_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_asin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_asinh_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_2d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bfloat16_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_block_diag_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_block_diag_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_byte_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ceil_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_chunk_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clamp_max_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clone_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clone_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_complex_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_complex_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cross_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cumulative_trapezoid_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diff_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_div_no_rounding_mode_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_div_no_rounding_mode_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_equal_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_equal_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_exp2_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_exp_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_copy_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_exponential_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fft2_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fft_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fft_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftn_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfft_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft2_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifftshift_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifftshift_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_irfft2_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_floor_divide_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmax_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_full_like_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gcd_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_geometric_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_half_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_heaviside_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_heaviside_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_hsplit_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_put_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_mean_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isfinite_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isneginf_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isposinf_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kron_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kthvalue_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kthvalue_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kthvalue_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_le_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_diagonal_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_eigvals_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_ldl_factor_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_ldl_solve_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lstsq_grad_oriented_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_slogdet_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_solve_triangular_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_tensorsolve_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log10_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log2_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_argmin_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumsum_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_select_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_softmax_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_reduction_no_dim_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_reduction_no_dim_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_reduction_with_dim_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_meshgrid_list_of_tensors_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_meshgrid_variadic_tensors_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_binary_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_reduction_no_dim_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_reduction_no_dim_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_min_reduction_with_dim_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mul_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mvlgamma_mvlgamma_p_3_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nan_to_num_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nanquantile_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_copy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_narrow_copy_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ne_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_neg_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_empty_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_full_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_ones_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_zeros_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_zeros_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_avg_pool3d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_avg_pool2d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_binary_cross_entropy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_dropout_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_feature_alpha_dropout_without_train_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_glu_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_hardtanh_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_huber_loss_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_huber_loss_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_trilinear_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_layer_norm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_logsigmoid_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_pool3d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_multi_head_attention_forward_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_normalize_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_constant_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_replicate_negative_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_shuffle_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_shuffle_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_rms_norm_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_rrelu_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_soft_margin_loss_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softmin_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_softsign_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_threshold_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_triplet_margin_loss_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_static_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_norm_fro_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_pca_lowrank_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_1_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_pow_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rand_like_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randint_like_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randn_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ravel_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_real_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_real_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resize_as__cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_conj_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_round_decimals_neg_3_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scalar_tensor_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_add_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_add_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_mean_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_scatter_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sgn_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_short_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sigmoid_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sign_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sin_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sinc_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sinh_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_softmax_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_softmax_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_softmax_with_dtype_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_airy_ai_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_t_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_u_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_hermite_polynomial_he_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_laguerre_polynomial_l_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_ndtr_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_ndtr_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_polygamma_special_polygamma_n_0_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_scaled_modified_bessel_k1_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_u_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_shifted_chebyshev_polynomial_v_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_list_args_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_split_with_sizes_copy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_std_mean_unbiased_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sum_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_t_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tensor_split_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tile_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_to_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_to_sparse_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapz_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapz_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapz_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_triu_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_true_divide_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_consecutive_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsafe_split_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsafe_split_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsafe_split_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsqueeze_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_vdot_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_as_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_view_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_vstack_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_where_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zeros_like_cuda_float64
2025-07-17T10:06:48.2786541Z 
2025-07-17T10:09:44.3842518Z 
2025-07-17T10:09:44.3843324Z inductor/test_torchinductor_opinfo 11/17 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_opinfo_11.17_b6fc59106bca3113_.log
2025-07-17T10:09:44.3938284Z Running 221 items in this shard: test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_T_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rand___cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rmatmul___cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rpow___cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive___rsub___cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__unsafe_masked_index_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive__unsafe_masked_index_put_accumulate_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_add_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addcmul_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_addcmul_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_amax_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_any_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_copy_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_partial_views_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_as_strided_scatter_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan2_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan2_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atan_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_atleast_2d_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bfloat16_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_block_diag_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_bmm_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_tensors_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_broadcast_to_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cauchy_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ceil_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cfloat_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_char_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_clone_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_conj_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_constant_pad_nd_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_contiguous_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cosh_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cosh_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cov_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cross_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cummin_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_cumsum_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diag_embed_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_diagonal_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_dist_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_einsum_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_empty_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expand_as_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expm1_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_expm1_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_fftshift_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfft_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_hfftn_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ifft_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fft_ihfft_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flatten_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flipud_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_flipud_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_float_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_fmax_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_full_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_full_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_full_like_cuda_uint16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ge_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ge_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_gt_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_i0_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_copy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_fill_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_put_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_amin_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_amin_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_index_reduce_prod_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_int_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isin_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isnan_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isneginf_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_isneginf_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_2inputs_2outputs_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_unary_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_jiterator_unary_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_kron_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lcm_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_ldexp_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_le_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lgamma_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_lgamma_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_cross_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_eigvals_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_lu_factor_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linalg_svd_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_linspace_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_log_softmax_with_dtype_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logaddexp_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logaddexp_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logdet_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_and_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_not_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logical_or_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logspace_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_logsumexp_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_cumprod_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_fill_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_mean_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_median_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_normalize_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_scatter_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_scatter_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_select_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_softmax_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_std_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_std_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_masked_sum_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_max_binary_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_maximum_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_maximum_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_maximum_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_maximum_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_meshgrid_variadic_tensors_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_movedim_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_msort_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mul_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_mul_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_native_dropout_backward_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_empty_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_empty_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_new_full_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nextafter_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_avg_pool2d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_max_pool1d_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_adaptive_max_pool2d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_avg_pool2d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_channel_shuffle_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_conv1d_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_cosine_embedding_loss_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_cosine_similarity_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_dropout2d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_embedding_bag_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_feature_alpha_dropout_without_train_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_fractional_max_pool3d_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_bicubic_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_interpolate_trilinear_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_local_response_norm_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_logsigmoid_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_max_unpool3d_grad_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_circular_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pad_replicate_negative_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pairwise_distance_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_shuffle_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_pixel_unshuffle_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_poisson_nll_loss_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_prelu_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_relu_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_threshold_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nn_functional_triplet_margin_with_distance_loss_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_nonzero_static_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_outer_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_permute_copy_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polar_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_0_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_1_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_3_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_4_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_polygamma_polygamma_n_4_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_prod_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_quantile_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rand_like_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randint_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_randint_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_real_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_repeat_interleave_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_reshape_as_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_reshape_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resize__cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_resolve_neg_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_roll_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rot90_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rot90_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_rsub_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_amax_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_amin_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_mean_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_scatter_reduce_sum_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_select_scatter_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sigmoid_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sign_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sinc_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_slice_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_slice_scatter_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_j1_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_bessel_j1_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_chebyshev_polynomial_t_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_hermite_polynomial_he_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i1_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_i1e_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_modified_bessel_i0_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_polygamma_special_polygamma_n_0_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_special_spherical_bessel_j0_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_square_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_stft_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_sum_to_size_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_take_along_dim_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tan_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_to_sparse_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapezoid_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_trapezoid_cuda_int32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tril_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_tril_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_triu_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_triu_cuda_uint8, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unbind_copy_cuda_float32, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unfold_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unique_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_unsafe_chunk_cuda_int64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_var_cuda_float16, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_var_cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_vstack_cuda_bool, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zero__cuda_float64, test/inductor/test_torchinductor_opinfo.py::TestInductorOpInfoCUDA::test_comprehensive_zeros_like_cuda_bool
2025-07-17T10:09:44.4021302Z 
2025-07-17T10:09:44.4021531Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:09:44.4021936Z Uploading artifacts took 0.00 seconds
2025-07-17T10:09:45.2957215Z Running test batch 'tests to run' cost 1320.55 seconds
2025-07-17T10:09:45.9150200Z + [[ 1 == 1 ]]
2025-07-17T10:09:45.9150832Z + [[ linux-jammy-rocm-py3.10 != linux-jammy-py3.9-gcc11-build ]]
2025-07-17T10:09:45.9152033Z + test_inductor_distributed
2025-07-17T10:09:45.9152667Z + echo 'Testing multi_gpu tests in test_torchinductor'
2025-07-17T10:09:45.9153524Z Testing multi_gpu tests in test_torchinductor
2025-07-17T10:09:45.9154364Z + python test/run_test.py -i inductor/test_aot_inductor.py -k test_replicate_on_devices --verbose
2025-07-17T10:09:48.9133947Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:09:48.9147776Z   import pkg_resources
2025-07-17T10:09:49.5756999Z No TD results found
2025-07-17T10:09:49.5758438Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:09:49.6006934Z Found test times from artifacts
2025-07-17T10:09:49.7038416Z Found test times from artifacts
2025-07-17T10:09:49.7077951Z Running all tests
2025-07-17T10:09:49.7083839Z Running parallel tests on 8 processes
2025-07-17T10:09:49.7084273Z Name: tests to run (est. time: 25.96min)
2025-07-17T10:09:49.7084534Z   Serial tests (0):
2025-07-17T10:09:49.7084753Z   Parallel tests (5):
2025-07-17T10:09:49.7084986Z     inductor/test_aot_inductor 1/5
2025-07-17T10:09:49.7085248Z     inductor/test_aot_inductor 2/5
2025-07-17T10:09:49.7085495Z     inductor/test_aot_inductor 3/5
2025-07-17T10:09:49.7085740Z     inductor/test_aot_inductor 4/5
2025-07-17T10:09:49.7085985Z     inductor/test_aot_inductor 5/5
2025-07-17T10:09:49.7086223Z Name: excluded (est. time: 0.0min)
2025-07-17T10:09:49.7086455Z   Serial tests (0):
2025-07-17T10:09:49.7086655Z   Parallel tests (0):
2025-07-17T10:09:49.7086974Z Running inductor/test_aot_inductor 1/5 ... [2025-07-17 10:09:49.708563]
2025-07-17T10:09:49.7087343Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:09:49.7090604Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=1', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:09:49.708884]
2025-07-17T10:09:57.1335216Z 
2025-07-17T10:09:57.1336591Z inductor/test_aot_inductor 1/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.5_4a90a8764905483c_.log
2025-07-17T10:09:57.1337850Z Running 0 items in this shard:
2025-07-17T10:09:57.1338131Z 
2025-07-17T10:09:57.1338512Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:09:57.1339206Z Uploading artifacts took 0.00 seconds
2025-07-17T10:09:57.1339823Z Running inductor/test_aot_inductor 2/5 ... [2025-07-17 10:09:57.133759]
2025-07-17T10:09:57.1340457Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:09:57.1343357Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=2', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:09:57.134056]
2025-07-17T10:10:04.5086059Z 
2025-07-17T10:10:04.5087181Z inductor/test_aot_inductor 2/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_2.5_f24953e011b99b02_.log
2025-07-17T10:10:04.5088413Z Running 0 items in this shard:
2025-07-17T10:10:04.5088697Z 
2025-07-17T10:10:04.5089033Z Running inductor/test_aot_inductor 3/5 ... [2025-07-17 10:10:04.508607]
2025-07-17T10:10:04.5089674Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:10:04.5092482Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=3', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:10:04.508918]
2025-07-17T10:10:11.9335107Z 
2025-07-17T10:10:11.9336249Z inductor/test_aot_inductor 3/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_3.5_cdc40adc6641178c_.log
2025-07-17T10:10:11.9337352Z Running 0 items in this shard:
2025-07-17T10:10:11.9337624Z 
2025-07-17T10:10:11.9337934Z Running inductor/test_aot_inductor 4/5 ... [2025-07-17 10:10:11.933458]
2025-07-17T10:10:11.9338752Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:10:11.9340368Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=4', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:10:11.933761]
2025-07-17T10:10:19.3582837Z 
2025-07-17T10:10:19.3584000Z inductor/test_aot_inductor 4/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_4.5_d7343af7d497f84d_.log
2025-07-17T10:10:19.3585017Z Running 0 items in this shard:
2025-07-17T10:10:19.3585184Z 
2025-07-17T10:10:19.3585418Z Running inductor/test_aot_inductor 5/5 ... [2025-07-17 10:10:19.358310]
2025-07-17T10:10:19.3585813Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:10:19.3591677Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=5', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:10:19.358841]
2025-07-17T10:10:26.7834981Z 
2025-07-17T10:10:26.7835903Z inductor/test_aot_inductor 5/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_5.5_c3de09a287917bbf_.log
2025-07-17T10:10:26.7836809Z Running 0 items in this shard:
2025-07-17T10:10:26.7837486Z 
2025-07-17T10:10:30.4453401Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:10:30.4455463Z   import pkg_resources
2025-07-17T10:10:30.5211880Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:10:30.5213389Z   import pkg_resources
2025-07-17T10:10:30.5310824Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:10:30.5312271Z   import pkg_resources
2025-07-17T10:10:30.5395908Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:10:30.5397995Z   import pkg_resources
2025-07-17T10:10:30.5420681Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:10:30.5422795Z   import pkg_resources
2025-07-17T10:10:30.5475734Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:10:30.5477880Z   import pkg_resources
2025-07-17T10:10:30.5580823Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:10:30.5583113Z   import pkg_resources
2025-07-17T10:10:30.5661588Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:10:30.5663566Z   import pkg_resources
2025-07-17T10:10:30.9678508Z Running inductor/test_aot_inductor 1/5 ... [2025-07-17 10:10:30.967595]
2025-07-17T10:10:30.9679031Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:10:30.9682438Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=1', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:10:30.968006]
2025-07-17T10:10:31.0798884Z Running inductor/test_aot_inductor 2/5 ... [2025-07-17 10:10:31.079653]
2025-07-17T10:10:31.0799411Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:10:31.0802806Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=2', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:10:31.080037]
2025-07-17T10:10:31.1281176Z Running inductor/test_aot_inductor 3/5 ... [2025-07-17 10:10:31.127861]
2025-07-17T10:10:31.1281680Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:10:31.1284844Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=3', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:10:31.128248]
2025-07-17T10:10:31.1396534Z Running inductor/test_aot_inductor 4/5 ... [2025-07-17 10:10:31.139496]
2025-07-17T10:10:31.1397057Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:10:31.1399252Z Running inductor/test_aot_inductor 5/5 ... [2025-07-17 10:10:31.139762]
2025-07-17T10:10:31.1399894Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:10:31.1401581Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=4', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:10:31.139859]
2025-07-17T10:10:31.1404443Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=5', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:10:31.140120]
2025-07-17T10:10:39.1549643Z 
2025-07-17T10:10:39.1551424Z inductor/test_aot_inductor 2/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_2.5_92ef939b7ed67456_.log
2025-07-17T10:10:39.1554039Z Running 2 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_replicate_on_devices_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_replicate_on_devices_cuda
2025-07-17T10:10:39.1555504Z 
2025-07-17T10:10:40.5555424Z 
2025-07-17T10:10:40.5556182Z inductor/test_aot_inductor 3/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_3.5_bfb2fc45f70480de_.log
2025-07-17T10:10:40.5557352Z Running 0 items in this shard:
2025-07-17T10:10:40.5557555Z 
2025-07-17T10:10:42.7196711Z 
2025-07-17T10:10:42.7197482Z inductor/test_aot_inductor 4/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_4.5_152a615351de7842_.log
2025-07-17T10:10:42.7198214Z Running 0 items in this shard:
2025-07-17T10:10:42.7198378Z 
2025-07-17T10:10:44.5506804Z 
2025-07-17T10:10:44.5507705Z inductor/test_aot_inductor 1/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.5_b8e9dc7bb3888390_.log
2025-07-17T10:10:44.5508618Z Running 0 items in this shard:
2025-07-17T10:10:44.5508835Z 
2025-07-17T10:10:46.3247987Z 
2025-07-17T10:10:46.3248951Z inductor/test_aot_inductor 5/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_5.5_98c0222248d132c5_.log
2025-07-17T10:10:46.3250908Z Running 1 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_replicate_on_devices_mps
2025-07-17T10:10:46.3251741Z 
2025-07-17T10:10:47.2394844Z Running test batch 'tests to run' cost 57.53 seconds
2025-07-17T10:10:47.8607804Z + python test/run_test.py -i inductor/test_aot_inductor.py -k test_on_gpu_device1 --verbose
2025-07-17T10:10:50.8369761Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:10:50.8372663Z   import pkg_resources
2025-07-17T10:10:51.4991992Z No TD results found
2025-07-17T10:10:51.4993320Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:10:51.5249468Z Found test times from artifacts
2025-07-17T10:10:51.6284664Z Found test times from artifacts
2025-07-17T10:10:51.6326321Z Running all tests
2025-07-17T10:10:51.6331553Z Running parallel tests on 8 processes
2025-07-17T10:10:51.6332181Z Name: tests to run (est. time: 25.96min)
2025-07-17T10:10:51.6332657Z   Serial tests (0):
2025-07-17T10:10:51.6333055Z   Parallel tests (5):
2025-07-17T10:10:51.6333456Z     inductor/test_aot_inductor 1/5
2025-07-17T10:10:51.6333917Z     inductor/test_aot_inductor 2/5
2025-07-17T10:10:51.6334363Z     inductor/test_aot_inductor 3/5
2025-07-17T10:10:51.6334825Z     inductor/test_aot_inductor 4/5
2025-07-17T10:10:51.6335242Z     inductor/test_aot_inductor 5/5
2025-07-17T10:10:51.6335672Z Name: excluded (est. time: 0.0min)
2025-07-17T10:10:51.6336210Z   Serial tests (0):
2025-07-17T10:10:51.6336570Z   Parallel tests (0):
2025-07-17T10:10:51.6336925Z Running inductor/test_aot_inductor 1/5 ... [2025-07-17 10:10:51.633303]
2025-07-17T10:10:51.6337308Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:10:51.6338433Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=1', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_on_gpu_device1', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:10:51.633623]
2025-07-17T10:10:59.1081881Z 
2025-07-17T10:10:59.1083642Z inductor/test_aot_inductor 1/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.5_8450398e4377f78c_.log
2025-07-17T10:10:59.1085100Z Running 0 items in this shard:
2025-07-17T10:10:59.1085400Z 
2025-07-17T10:10:59.1085784Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:10:59.1086479Z Uploading artifacts took 0.00 seconds
2025-07-17T10:10:59.1087094Z Running inductor/test_aot_inductor 2/5 ... [2025-07-17 10:10:59.108142]
2025-07-17T10:10:59.1087843Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:10:59.1090094Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=2', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_on_gpu_device1', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:10:59.108439]
2025-07-17T10:11:06.4829951Z 
2025-07-17T10:11:06.4831260Z inductor/test_aot_inductor 2/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_2.5_94e16a25228fc0e2_.log
2025-07-17T10:11:06.4832189Z Running 0 items in this shard:
2025-07-17T10:11:06.4832405Z 
2025-07-17T10:11:06.4832674Z Running inductor/test_aot_inductor 3/5 ... [2025-07-17 10:11:06.482854]
2025-07-17T10:11:06.4833151Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:11:06.4834380Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=3', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_on_gpu_device1', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:11:06.483145]
2025-07-17T10:11:13.9577635Z 
2025-07-17T10:11:13.9578815Z inductor/test_aot_inductor 3/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_3.5_991a4bb479ce636e_.log
2025-07-17T10:11:13.9580238Z Running 0 items in this shard:
2025-07-17T10:11:13.9580597Z 
2025-07-17T10:11:13.9580991Z Running inductor/test_aot_inductor 4/5 ... [2025-07-17 10:11:13.957754]
2025-07-17T10:11:13.9582504Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:11:13.9585145Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=4', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_on_gpu_device1', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:11:13.958266]
2025-07-17T10:11:21.3828838Z 
2025-07-17T10:11:21.3829955Z inductor/test_aot_inductor 4/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_4.5_b796b5a0839670b7_.log
2025-07-17T10:11:21.3831387Z Running 0 items in this shard:
2025-07-17T10:11:21.3831723Z 
2025-07-17T10:11:21.3832092Z Running inductor/test_aot_inductor 5/5 ... [2025-07-17 10:11:21.382814]
2025-07-17T10:11:21.3832853Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:11:21.3834078Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=5', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_on_gpu_device1', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:11:21.383175]
2025-07-17T10:11:29.3080634Z 
2025-07-17T10:11:29.3082022Z inductor/test_aot_inductor 5/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_5.5_dff73a946ff8081e_.log
2025-07-17T10:11:29.3083377Z Running 0 items in this shard:
2025-07-17T10:11:29.3083666Z 
2025-07-17T10:11:32.9051529Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:11:32.9054230Z   import pkg_resources
2025-07-17T10:11:32.9429822Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:11:32.9431318Z   import pkg_resources
2025-07-17T10:11:32.9902061Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:11:32.9903703Z   import pkg_resources
2025-07-17T10:11:33.0551389Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:11:33.0553386Z   import pkg_resources
2025-07-17T10:11:33.0665876Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:11:33.0667913Z   import pkg_resources
2025-07-17T10:11:33.0673519Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:11:33.0674816Z   import pkg_resources
2025-07-17T10:11:33.0684793Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:11:33.0686220Z   import pkg_resources
2025-07-17T10:11:33.0758129Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:11:33.0760090Z   import pkg_resources
2025-07-17T10:11:33.3720798Z Running inductor/test_aot_inductor 1/5 ... [2025-07-17 10:11:33.371875]
2025-07-17T10:11:33.3721350Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:11:33.3724902Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=1', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_on_gpu_device1', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:11:33.372259]
2025-07-17T10:11:33.4521099Z Running inductor/test_aot_inductor 2/5 ... [2025-07-17 10:11:33.451859]
2025-07-17T10:11:33.4521599Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:11:33.4525487Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=2', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_on_gpu_device1', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:11:33.452307]
2025-07-17T10:11:33.5000003Z Running inductor/test_aot_inductor 3/5 ... [2025-07-17 10:11:33.499787]
2025-07-17T10:11:33.5001060Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:11:33.5004457Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=3', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_on_gpu_device1', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:11:33.500153]
2025-07-17T10:11:33.5835959Z Running inductor/test_aot_inductor 4/5 ... [2025-07-17 10:11:33.583404]
2025-07-17T10:11:33.5836827Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:11:33.5840467Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=4', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_on_gpu_device1', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:11:33.583772]
2025-07-17T10:11:33.6194894Z Running inductor/test_aot_inductor 5/5 ... [2025-07-17 10:11:33.619300]
2025-07-17T10:11:33.6195364Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:11:33.6198903Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=5', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_on_gpu_device1', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:11:33.619671]
2025-07-17T10:11:41.0473418Z 
2025-07-17T10:11:41.0474572Z inductor/test_aot_inductor 1/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.5_270831c9292df168_.log
2025-07-17T10:11:41.0475814Z Running 0 items in this shard:
2025-07-17T10:11:41.0476103Z 
2025-07-17T10:11:42.8296038Z 
2025-07-17T10:11:42.8297172Z inductor/test_aot_inductor 2/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_2.5_b9f3bad914e5c455_.log
2025-07-17T10:11:42.8298931Z Running 2 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_on_gpu_device1_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_on_gpu_device1_cuda
2025-07-17T10:11:42.8300313Z 
2025-07-17T10:11:44.6627798Z 
2025-07-17T10:11:44.6628654Z inductor/test_aot_inductor 4/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_4.5_6f9a9c4a3107ca8b_.log
2025-07-17T10:11:44.6629557Z Running 0 items in this shard:
2025-07-17T10:11:44.6629780Z 
2025-07-17T10:11:46.5321257Z 
2025-07-17T10:11:46.5322334Z inductor/test_aot_inductor 3/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_3.5_0a756facd3b8f06e_.log
2025-07-17T10:11:46.5323545Z Running 0 items in this shard:
2025-07-17T10:11:46.5323835Z 
2025-07-17T10:11:48.6536184Z 
2025-07-17T10:11:48.6537306Z inductor/test_aot_inductor 5/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_5.5_509499fc4b09f2f5_.log
2025-07-17T10:11:48.6539043Z Running 1 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_on_gpu_device1_mps
2025-07-17T10:11:48.6539826Z 
2025-07-17T10:11:49.5501001Z Running test batch 'tests to run' cost 57.92 seconds
2025-07-17T10:11:50.1548353Z + python test/run_test.py -i inductor/test_aot_inductor.py -k test_non_default_gpu_device --verbose
2025-07-17T10:11:53.1606150Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:11:53.1608276Z   import pkg_resources
2025-07-17T10:11:53.8194498Z No TD results found
2025-07-17T10:11:53.8196536Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:11:53.8443208Z Found test times from artifacts
2025-07-17T10:11:53.9469985Z Found test times from artifacts
2025-07-17T10:11:53.9509433Z Running all tests
2025-07-17T10:11:53.9513411Z Running parallel tests on 8 processes
2025-07-17T10:11:53.9514115Z Name: tests to run (est. time: 25.96min)
2025-07-17T10:11:53.9514565Z   Serial tests (0):
2025-07-17T10:11:53.9514937Z   Parallel tests (5):
2025-07-17T10:11:53.9515326Z     inductor/test_aot_inductor 1/5
2025-07-17T10:11:53.9516027Z     inductor/test_aot_inductor 2/5
2025-07-17T10:11:53.9516440Z     inductor/test_aot_inductor 3/5
2025-07-17T10:11:53.9516851Z     inductor/test_aot_inductor 4/5
2025-07-17T10:11:53.9517242Z     inductor/test_aot_inductor 5/5
2025-07-17T10:11:53.9517488Z Name: excluded (est. time: 0.0min)
2025-07-17T10:11:53.9517711Z   Serial tests (0):
2025-07-17T10:11:53.9517916Z   Parallel tests (0):
2025-07-17T10:11:53.9518245Z Running inductor/test_aot_inductor 1/5 ... [2025-07-17 10:11:53.951500]
2025-07-17T10:11:53.9518617Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:11:53.9519809Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=1', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_gpu_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:11:53.951814]
2025-07-17T10:12:01.4264221Z 
2025-07-17T10:12:01.4265257Z inductor/test_aot_inductor 1/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.5_e8fdbe5b5f08317f_.log
2025-07-17T10:12:01.4266451Z Running 0 items in this shard:
2025-07-17T10:12:01.4266724Z 
2025-07-17T10:12:01.4267099Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:12:01.4267795Z Uploading artifacts took 0.00 seconds
2025-07-17T10:12:01.4268535Z Running inductor/test_aot_inductor 2/5 ... [2025-07-17 10:12:01.426617]
2025-07-17T10:12:01.4269280Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:12:01.4272571Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=2', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_gpu_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:12:01.427009]
2025-07-17T10:12:08.8514734Z 
2025-07-17T10:12:08.8515936Z inductor/test_aot_inductor 2/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_2.5_30297fb53af959d3_.log
2025-07-17T10:12:08.8517174Z Running 0 items in this shard:
2025-07-17T10:12:08.8517464Z 
2025-07-17T10:12:08.8517798Z Running inductor/test_aot_inductor 3/5 ... [2025-07-17 10:12:08.851365]
2025-07-17T10:12:08.8518449Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:12:08.8519879Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=3', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_gpu_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:12:08.851665]
2025-07-17T10:12:16.2762203Z 
2025-07-17T10:12:16.2763352Z inductor/test_aot_inductor 3/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_3.5_5c68baee9dd5da66_.log
2025-07-17T10:12:16.2764564Z Running 0 items in this shard:
2025-07-17T10:12:16.2764868Z 
2025-07-17T10:12:16.2765196Z Running inductor/test_aot_inductor 4/5 ... [2025-07-17 10:12:16.276103]
2025-07-17T10:12:16.2765824Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:12:16.2768402Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=4', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_gpu_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:12:16.276403]
2025-07-17T10:12:23.7008744Z 
2025-07-17T10:12:23.7010535Z inductor/test_aot_inductor 4/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_4.5_a67851d627f8cb15_.log
2025-07-17T10:12:23.7011939Z Running 0 items in this shard:
2025-07-17T10:12:23.7012268Z 
2025-07-17T10:12:23.7012666Z Running inductor/test_aot_inductor 5/5 ... [2025-07-17 10:12:23.700822]
2025-07-17T10:12:23.7013673Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:12:23.7016233Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=5', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_gpu_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:12:23.701336]
2025-07-17T10:12:31.1756366Z 
2025-07-17T10:12:31.1757406Z inductor/test_aot_inductor 5/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_5.5_98230e4194da74f3_.log
2025-07-17T10:12:31.1758582Z Running 0 items in this shard:
2025-07-17T10:12:31.1758854Z 
2025-07-17T10:12:34.9679950Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:12:34.9681533Z   import pkg_resources
2025-07-17T10:12:35.0038583Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:12:35.0040621Z   import pkg_resources
2025-07-17T10:12:35.0158597Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:12:35.0161223Z   import pkg_resources
2025-07-17T10:12:35.0224720Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:12:35.0226769Z   import pkg_resources
2025-07-17T10:12:35.0240596Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:12:35.0241832Z   import pkg_resources
2025-07-17T10:12:35.0277774Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:12:35.0317324Z   import pkg_resources
2025-07-17T10:12:35.0319228Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:12:35.0321195Z   import pkg_resources
2025-07-17T10:12:35.0852537Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:12:35.0854579Z   import pkg_resources
2025-07-17T10:12:35.4878566Z Running inductor/test_aot_inductor 1/5 ... [2025-07-17 10:12:35.487643]
2025-07-17T10:12:35.4879276Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:12:35.4882645Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=1', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_gpu_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:12:35.488031]
2025-07-17T10:12:35.5924053Z Running inductor/test_aot_inductor 2/5 ... [2025-07-17 10:12:35.592228]
2025-07-17T10:12:35.5924536Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:12:35.5928652Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=2', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_gpu_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:12:35.592651]
2025-07-17T10:12:35.6034541Z Running inductor/test_aot_inductor 3/5 ... [2025-07-17 10:12:35.603295]
2025-07-17T10:12:35.6035026Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:12:35.6038818Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=3', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_gpu_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:12:35.603662]
2025-07-17T10:12:35.6198129Z Running inductor/test_aot_inductor 4/5 ... [2025-07-17 10:12:35.619633]
2025-07-17T10:12:35.6198800Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:12:35.6199237Z Running inductor/test_aot_inductor 5/5 ... [2025-07-17 10:12:35.619754]
2025-07-17T10:12:35.6199676Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:12:35.6202688Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=4', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_gpu_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:12:35.620054]
2025-07-17T10:12:35.6204780Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=5', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_gpu_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:12:35.620166]
2025-07-17T10:12:43.1174680Z 
2025-07-17T10:12:43.1175577Z inductor/test_aot_inductor 2/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_2.5_f7720cb00fde10b9_.log
2025-07-17T10:12:43.1179245Z Running 1 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_non_default_gpu_device_mps
2025-07-17T10:12:43.1179902Z 
2025-07-17T10:12:45.3975415Z 
2025-07-17T10:12:45.3976606Z inductor/test_aot_inductor 4/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_4.5_8cc62f70a26f099a_.log
2025-07-17T10:12:45.3978996Z Running 2 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_non_default_gpu_device_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_non_default_gpu_device_cuda
2025-07-17T10:12:45.3980437Z 
2025-07-17T10:12:46.7673993Z 
2025-07-17T10:12:46.7675495Z inductor/test_aot_inductor 1/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.5_188b4348043a1a6f_.log
2025-07-17T10:12:46.7676916Z Running 0 items in this shard:
2025-07-17T10:12:46.7677204Z 
2025-07-17T10:12:48.7520714Z 
2025-07-17T10:12:48.7521784Z inductor/test_aot_inductor 5/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_5.5_1d7449b1b39fa7b5_.log
2025-07-17T10:12:48.7523018Z Running 0 items in this shard:
2025-07-17T10:12:48.7523963Z 
2025-07-17T10:12:50.8888310Z 
2025-07-17T10:12:50.8889448Z inductor/test_aot_inductor 3/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_3.5_4c8447d5269e5af8_.log
2025-07-17T10:12:50.8894627Z Running 0 items in this shard:
2025-07-17T10:12:50.8894966Z 
2025-07-17T10:12:51.7439848Z Running test batch 'tests to run' cost 57.79 seconds
2025-07-17T10:12:52.3621415Z + python test/run_test.py -i inductor/test_aot_inductor.py -k test_load_package_multiple_gpus --verbose
2025-07-17T10:12:55.3243536Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:12:55.3245618Z   import pkg_resources
2025-07-17T10:12:55.9790453Z No TD results found
2025-07-17T10:12:55.9791789Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:12:56.0044157Z Found test times from artifacts
2025-07-17T10:12:56.1077118Z Found test times from artifacts
2025-07-17T10:12:56.1118719Z Running all tests
2025-07-17T10:12:56.1129302Z Running parallel tests on 8 processes
2025-07-17T10:12:56.1129982Z Name: tests to run (est. time: 25.96min)
2025-07-17T10:12:56.1130252Z   Serial tests (0):
2025-07-17T10:12:56.1130457Z   Parallel tests (5):
2025-07-17T10:12:56.1130989Z     inductor/test_aot_inductor 1/5
2025-07-17T10:12:56.1131261Z     inductor/test_aot_inductor 2/5
2025-07-17T10:12:56.1131513Z     inductor/test_aot_inductor 3/5
2025-07-17T10:12:56.1131755Z     inductor/test_aot_inductor 4/5
2025-07-17T10:12:56.1132001Z     inductor/test_aot_inductor 5/5
2025-07-17T10:12:56.1132247Z Name: excluded (est. time: 0.0min)
2025-07-17T10:12:56.1132484Z   Serial tests (0):
2025-07-17T10:12:56.1132689Z   Parallel tests (0):
2025-07-17T10:12:56.1133009Z Running inductor/test_aot_inductor 1/5 ... [2025-07-17 10:12:56.113143]
2025-07-17T10:12:56.1133391Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:12:56.1136455Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=1', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_load_package_multiple_gpus', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:12:56.113463]
2025-07-17T10:13:03.5882342Z 
2025-07-17T10:13:03.5883464Z inductor/test_aot_inductor 1/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.5_b75cc40b36b52546_.log
2025-07-17T10:13:03.5884689Z Running 0 items in this shard:
2025-07-17T10:13:03.5884979Z 
2025-07-17T10:13:03.5885365Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:13:03.5886180Z Uploading artifacts took 0.00 seconds
2025-07-17T10:13:03.5886941Z Running inductor/test_aot_inductor 2/5 ... [2025-07-17 10:13:03.588462]
2025-07-17T10:13:03.5887686Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:13:03.5891976Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=2', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_load_package_multiple_gpus', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:13:03.588972]
2025-07-17T10:13:11.0135258Z 
2025-07-17T10:13:11.0136290Z inductor/test_aot_inductor 2/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_2.5_3ad82210c979d31a_.log
2025-07-17T10:13:11.0137224Z Running 0 items in this shard:
2025-07-17T10:13:11.0137500Z 
2025-07-17T10:13:11.0137849Z Running inductor/test_aot_inductor 3/5 ... [2025-07-17 10:13:11.013504]
2025-07-17T10:13:11.0138515Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:13:11.0143386Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=3', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_load_package_multiple_gpus', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:13:11.014052]
2025-07-17T10:13:18.4384180Z 
2025-07-17T10:13:18.4385417Z inductor/test_aot_inductor 3/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_3.5_dcf794626685cefd_.log
2025-07-17T10:13:18.4386639Z Running 0 items in this shard:
2025-07-17T10:13:18.4386939Z 
2025-07-17T10:13:18.4387328Z Running inductor/test_aot_inductor 4/5 ... [2025-07-17 10:13:18.438385]
2025-07-17T10:13:18.4388105Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:13:18.4391695Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=4', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_load_package_multiple_gpus', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:13:18.438875]
2025-07-17T10:13:25.8633007Z 
2025-07-17T10:13:25.8633795Z inductor/test_aot_inductor 4/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_4.5_4c9c11ec02d46138_.log
2025-07-17T10:13:25.8634541Z Running 0 items in this shard:
2025-07-17T10:13:25.8634732Z 
2025-07-17T10:13:25.8634945Z Running inductor/test_aot_inductor 5/5 ... [2025-07-17 10:13:25.863189]
2025-07-17T10:13:25.8635820Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:13:25.8636928Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'serial', '--shard-id=5', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_load_package_multiple_gpus', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:13:25.863482]
2025-07-17T10:13:33.7384269Z 
2025-07-17T10:13:33.7385455Z inductor/test_aot_inductor 5/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_5.5_1f5480b00988879d_.log
2025-07-17T10:13:33.7386611Z Running 0 items in this shard:
2025-07-17T10:13:33.7386918Z 
2025-07-17T10:13:37.4064476Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:13:37.4066615Z   import pkg_resources
2025-07-17T10:13:37.4120384Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:13:37.4122392Z   import pkg_resources
2025-07-17T10:13:37.4518511Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:13:37.4521078Z   import pkg_resources
2025-07-17T10:13:37.4689076Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:13:37.4691072Z   import pkg_resources
2025-07-17T10:13:37.4692989Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:13:37.4695141Z   import pkg_resources
2025-07-17T10:13:37.4965903Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:13:37.4967981Z   import pkg_resources
2025-07-17T10:13:37.5009170Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:13:37.5011174Z   import pkg_resources
2025-07-17T10:13:37.5318894Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:13:37.5320869Z   import pkg_resources
2025-07-17T10:13:37.9517799Z Running inductor/test_aot_inductor 1/5 ... [2025-07-17 10:13:37.951598]
2025-07-17T10:13:37.9518742Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:13:37.9522340Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=1', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_load_package_multiple_gpus', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:13:37.951969]
2025-07-17T10:13:37.9883801Z Running inductor/test_aot_inductor 2/5 ... [2025-07-17 10:13:37.988207]
2025-07-17T10:13:37.9884462Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:13:37.9888690Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=2', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_load_package_multiple_gpus', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:13:37.988636]
2025-07-17T10:13:38.0445805Z Running inductor/test_aot_inductor 3/5 ... [2025-07-17 10:13:38.044414]
2025-07-17T10:13:38.0446432Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:13:38.0450459Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=3', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_load_package_multiple_gpus', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:13:38.044834]
2025-07-17T10:13:38.0838892Z Running inductor/test_aot_inductor 4/5 ... [2025-07-17 10:13:38.083740]
2025-07-17T10:13:38.0839255Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:13:38.0841682Z Running inductor/test_aot_inductor 5/5 ... [2025-07-17 10:13:38.084009]
2025-07-17T10:13:38.0842282Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:13:38.0844500Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=4', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_load_package_multiple_gpus', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:13:38.084151]
2025-07-17T10:13:38.0847485Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '-m', 'not serial', '--shard-id=5', '--num-shards=5', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_load_package_multiple_gpus', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:13:38.084422]
2025-07-17T10:13:45.6267571Z 
2025-07-17T10:13:45.6268409Z inductor/test_aot_inductor 1/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.5_3f6497a58eb01e3f_.log
2025-07-17T10:13:45.6269497Z Running 1 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_load_package_multiple_gpus_cpu
2025-07-17T10:13:45.6270009Z 
2025-07-17T10:13:47.8122049Z 
2025-07-17T10:13:47.8123198Z inductor/test_aot_inductor 5/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_5.5_d18f8d69001bff41_.log
2025-07-17T10:13:47.8124545Z Running 0 items in this shard:
2025-07-17T10:13:47.8124886Z 
2025-07-17T10:13:49.2131339Z 
2025-07-17T10:13:49.2132554Z inductor/test_aot_inductor 4/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_4.5_a4be74fed83c2fce_.log
2025-07-17T10:13:49.2134400Z Running 1 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_load_package_multiple_gpus_cuda
2025-07-17T10:13:49.2135260Z 
2025-07-17T10:13:51.1209679Z 
2025-07-17T10:13:51.1210843Z inductor/test_aot_inductor 2/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_2.5_3718f9c90b5fb29f_.log
2025-07-17T10:13:51.1212020Z Running 0 items in this shard:
2025-07-17T10:13:51.1212748Z 
2025-07-17T10:13:52.9294494Z 
2025-07-17T10:13:52.9295781Z inductor/test_aot_inductor 3/5 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_3.5_d66002e83aa0c81d_.log
2025-07-17T10:13:52.9297705Z Running 1 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleMps::test_load_package_multiple_gpus_mps
2025-07-17T10:13:52.9298565Z 
2025-07-17T10:13:53.8004148Z Running test batch 'tests to run' cost 57.69 seconds
2025-07-17T10:13:54.4095115Z + python test/run_test.py -i distributed/test_c10d_functional_native.py --verbose
2025-07-17T10:13:57.4064680Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:13:57.4067105Z   import pkg_resources
2025-07-17T10:13:58.0633710Z No TD results found
2025-07-17T10:13:58.0634990Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:13:58.0883525Z Found test times from artifacts
2025-07-17T10:13:58.1922173Z Found test times from artifacts
2025-07-17T10:13:58.1964363Z Running all tests
2025-07-17T10:13:58.1968194Z Running parallel tests on 8 processes
2025-07-17T10:13:58.1968729Z Name: tests to run (est. time: 2.93min)
2025-07-17T10:13:58.1969172Z   Serial tests (1):
2025-07-17T10:13:58.1969591Z     distributed/test_c10d_functional_native 1/1
2025-07-17T10:13:58.1970080Z   Parallel tests (0):
2025-07-17T10:13:58.1970535Z Name: excluded (est. time: 0.0min)
2025-07-17T10:13:58.1971039Z   Serial tests (0):
2025-07-17T10:13:58.1971451Z   Parallel tests (0):
2025-07-17T10:13:58.1972679Z Running distributed/test_c10d_functional_native 1/1 ... [2025-07-17 10:13:58.196940]
2025-07-17T10:13:58.1973563Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:13:58.1975028Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:13:58.197253]
2025-07-17T10:17:21.1236593Z 
2025-07-17T10:17:21.1237702Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_c0ede5a60ded27d5_.log
2025-07-17T10:17:21.1251908Z Running 32 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_fixed_striding, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_functional_collectives_inference_mode, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_wait_tensor, test/distributed/test_c10d_functional_native.py::PyWorkTest::test_collectives, test/distributed/test_c10d_functional_native.py::PyWorkTest::test_wait_tensor, test/distributed/test_c10d_functional_native.py::CompileTestCPU::test_inductor_all_reduce_cpu, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_non_contig_input, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag, test/distributed/test_c10d_functional_native.py::CompileTest::test_wait_tensor
2025-07-17T10:17:21.1262235Z 
2025-07-17T10:17:21.1262460Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:17:21.1262880Z Uploading artifacts took 0.00 seconds
2025-07-17T10:17:24.7863062Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:17:24.7865250Z   import pkg_resources
2025-07-17T10:17:24.8682099Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:17:24.8684347Z   import pkg_resources
2025-07-17T10:17:24.8727829Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:17:24.8729817Z   import pkg_resources
2025-07-17T10:17:24.8766663Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:17:24.8768662Z   import pkg_resources
2025-07-17T10:17:24.8770567Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:17:24.8772524Z   import pkg_resources
2025-07-17T10:17:24.8796669Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:17:24.8798879Z   import pkg_resources
2025-07-17T10:17:24.8876346Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:17:24.8878280Z   import pkg_resources
2025-07-17T10:17:24.9163859Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:17:24.9165910Z   import pkg_resources
2025-07-17T10:17:26.3960404Z Running test batch 'tests to run' cost 208.2 seconds
2025-07-17T10:17:27.0075109Z + python test/run_test.py -i distributed/tensor/test_dtensor_compile.py --verbose
2025-07-17T10:17:29.9965750Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:17:29.9967908Z   import pkg_resources
2025-07-17T10:17:30.6527866Z No TD results found
2025-07-17T10:17:30.6529380Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:17:30.6781058Z Found test times from artifacts
2025-07-17T10:17:30.7812991Z Found test times from artifacts
2025-07-17T10:17:30.7855739Z Running all tests
2025-07-17T10:17:30.7858640Z Running parallel tests on 8 processes
2025-07-17T10:17:30.7859254Z Name: tests to run (est. time: 2.72min)
2025-07-17T10:17:30.7860063Z   Serial tests (1):
2025-07-17T10:17:30.7860572Z     distributed/tensor/test_dtensor_compile 1/1
2025-07-17T10:17:30.7861162Z   Parallel tests (0):
2025-07-17T10:17:30.7861624Z Name: excluded (est. time: 0.0min)
2025-07-17T10:17:30.7861939Z   Serial tests (0):
2025-07-17T10:17:30.7862168Z   Parallel tests (0):
2025-07-17T10:17:30.7862687Z Running distributed/tensor/test_dtensor_compile 1/1 ... [2025-07-17 10:17:30.785970]
2025-07-17T10:17:30.7863103Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:17:30.7864584Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:17:30.786291]
2025-07-17T10:21:17.1440950Z 
2025-07-17T10:21:17.1442377Z distributed/tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dtensor_compile_1.1_af7f15b2227df5ba_.log
2025-07-17T10:21:17.1462962Z Running 40 items in this shard: test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_basic, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_basic_export, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_cat, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_slice, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_dynamic_shapes, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_async, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering_graph_partition, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_unwrap_async_collective_tensor_tangent, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_ca_True
2025-07-17T10:21:17.1477940Z 
2025-07-17T10:21:17.1478160Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:21:17.1478569Z Uploading artifacts took 0.00 seconds
2025-07-17T10:21:20.8635945Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:20.8637961Z   import pkg_resources
2025-07-17T10:21:20.8788386Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:20.8789868Z   import pkg_resources
2025-07-17T10:21:20.9194120Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:20.9202580Z   import pkg_resources
2025-07-17T10:21:20.9471995Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:20.9473184Z   import pkg_resources
2025-07-17T10:21:20.9565863Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:20.9567305Z   import pkg_resources
2025-07-17T10:21:20.9595250Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:20.9596823Z   import pkg_resources
2025-07-17T10:21:20.9637064Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:20.9638659Z   import pkg_resources
2025-07-17T10:21:20.9681878Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:20.9683353Z   import pkg_resources
2025-07-17T10:21:22.4716106Z Running test batch 'tests to run' cost 231.69 seconds
2025-07-17T10:21:23.0943922Z + python test/run_test.py -i distributed/tensor/parallel/test_micro_pipeline_tp.py --verbose
2025-07-17T10:21:26.0946640Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:26.0949196Z   import pkg_resources
2025-07-17T10:21:26.7554223Z No TD results found
2025-07-17T10:21:26.7555319Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:21:26.7803117Z Found test times from artifacts
2025-07-17T10:21:26.8838307Z Found test times from artifacts
2025-07-17T10:21:26.8880186Z Running all tests
2025-07-17T10:21:26.8883913Z Running parallel tests on 8 processes
2025-07-17T10:21:26.8884440Z Name: tests to run (est. time: 0.24min)
2025-07-17T10:21:26.8885341Z   Serial tests (1):
2025-07-17T10:21:26.8885825Z     distributed/tensor/parallel/test_micro_pipeline_tp 1/1
2025-07-17T10:21:26.8886396Z   Parallel tests (0):
2025-07-17T10:21:26.8886782Z Name: excluded (est. time: 0.0min)
2025-07-17T10:21:26.8887201Z   Serial tests (0):
2025-07-17T10:21:26.8887547Z   Parallel tests (0):
2025-07-17T10:21:26.8888245Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2025-07-17 10:21:26.888504]
2025-07-17T10:21:26.8889046Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:21:26.8890768Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:21:26.888833]
2025-07-17T10:21:49.5338483Z 
2025-07-17T10:21:49.5339756Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_90a4a97b647d1fce_.log
2025-07-17T10:21:49.5370913Z Running 44 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_rowwise_scales_reshape_mm_reshape_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_rowwise_scales_reshape_mm_reshape_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_rowwise_scales_reshape_mm_reshape_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives
2025-07-17T10:21:49.5392002Z 
2025-07-17T10:21:49.5392234Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:21:49.5392651Z Uploading artifacts took 0.00 seconds
2025-07-17T10:21:53.1405222Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:53.1407420Z   import pkg_resources
2025-07-17T10:21:53.1408877Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:53.1410184Z   import pkg_resources
2025-07-17T10:21:53.1933717Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:53.1935164Z   import pkg_resources
2025-07-17T10:21:53.2113378Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:53.2114838Z   import pkg_resources
2025-07-17T10:21:53.2281585Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:53.2283026Z   import pkg_resources
2025-07-17T10:21:53.2323012Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:53.2324488Z   import pkg_resources
2025-07-17T10:21:53.2326048Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:53.2327481Z   import pkg_resources
2025-07-17T10:21:53.2468669Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:53.2470107Z   import pkg_resources
2025-07-17T10:21:54.5757715Z Running test batch 'tests to run' cost 27.69 seconds
2025-07-17T10:21:55.1885983Z + python test/run_test.py -i distributed/_composable/test_replicate_with_compiler.py --verbose
2025-07-17T10:21:58.1925266Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:21:58.1927370Z   import pkg_resources
2025-07-17T10:21:58.8561464Z No TD results found
2025-07-17T10:21:58.8563108Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:21:58.8810668Z Found test times from artifacts
2025-07-17T10:21:58.9842169Z Found test times from artifacts
2025-07-17T10:21:58.9883428Z Running all tests
2025-07-17T10:21:58.9887157Z Running parallel tests on 8 processes
2025-07-17T10:21:58.9887684Z Name: tests to run (est. time: 2.54min)
2025-07-17T10:21:58.9888141Z   Serial tests (1):
2025-07-17T10:21:58.9889041Z     distributed/_composable/test_replicate_with_compiler 1/1
2025-07-17T10:21:58.9889670Z   Parallel tests (0):
2025-07-17T10:21:58.9890137Z Name: excluded (est. time: 0.0min)
2025-07-17T10:21:58.9890636Z   Serial tests (0):
2025-07-17T10:21:58.9891051Z   Parallel tests (0):
2025-07-17T10:21:58.9891882Z Running distributed/_composable/test_replicate_with_compiler 1/1 ... [2025-07-17 10:21:58.988835]
2025-07-17T10:21:58.9892823Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:21:58.9894208Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate_with_compiler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:21:58.989158]
2025-07-17T10:24:52.4868025Z 
2025-07-17T10:24:52.4869904Z distributed/_composable/test_replicate_with_compiler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_b793abfacba4fd8c_.log
2025-07-17T10:24:52.4874924Z Running 10 items in this shard: test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_coalesced_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_concat_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_backward_only, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_bf16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu_no_sync, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_fp16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu_ac, test/distributed/_composable/test_replicate_with_compiler.py::DDP_TP_Test::test_ddp_tp
2025-07-17T10:24:52.4878731Z 
2025-07-17T10:24:52.4878975Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:24:52.4879404Z Uploading artifacts took 0.00 seconds
2025-07-17T10:24:56.0923186Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:24:56.0925938Z   import pkg_resources
2025-07-17T10:24:56.1266079Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:24:56.1268148Z   import pkg_resources
2025-07-17T10:24:56.1377714Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:24:56.1379708Z   import pkg_resources
2025-07-17T10:24:56.2057748Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:24:56.2059769Z   import pkg_resources
2025-07-17T10:24:56.2121006Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:24:56.2123266Z   import pkg_resources
2025-07-17T10:24:56.2163466Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:24:56.2165476Z   import pkg_resources
2025-07-17T10:24:56.2383611Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:24:56.2385615Z   import pkg_resources
2025-07-17T10:24:56.2583693Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:24:56.2585679Z   import pkg_resources
2025-07-17T10:24:57.5293041Z Running test batch 'tests to run' cost 178.54 seconds
2025-07-17T10:24:58.1373384Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_comm.py --verbose
2025-07-17T10:25:01.1690480Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:25:01.1692671Z   import pkg_resources
2025-07-17T10:25:01.8313594Z No TD results found
2025-07-17T10:25:01.8315070Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:25:01.8569380Z Found test times from artifacts
2025-07-17T10:25:01.9598531Z Found test times from artifacts
2025-07-17T10:25:01.9639085Z Running all tests
2025-07-17T10:25:01.9642948Z Running parallel tests on 8 processes
2025-07-17T10:25:01.9643468Z Name: tests to run (est. time: 3.33min)
2025-07-17T10:25:01.9643908Z   Serial tests (1):
2025-07-17T10:25:01.9644367Z     distributed/_composable/fsdp/test_fully_shard_comm 1/1
2025-07-17T10:25:01.9644909Z   Parallel tests (0):
2025-07-17T10:25:01.9645291Z Name: excluded (est. time: 0.0min)
2025-07-17T10:25:01.9651877Z   Serial tests (0):
2025-07-17T10:25:01.9652089Z   Parallel tests (0):
2025-07-17T10:25:01.9652492Z Running distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2025-07-17 10:25:01.964417]
2025-07-17T10:25:01.9652955Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:25:01.9653932Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:25:01.964767]
2025-07-17T10:28:25.3386672Z 
2025-07-17T10:28:25.3388361Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_0ef971839f1a90c8_.log
2025-07-17T10:28:25.3409645Z Running 21 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_all_gather_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp16, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_fully_shard_communication_count, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_manual_reshard_with_reshard_after_forward_false, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_set_reduce_scatter_divide_factor, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_set_reshard_after_forward, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_backward_misprefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_unused_module, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_forward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiProcess::test_unshard_async, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_no_param_group, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_without_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardAllocFromPG::test_exception_when_used_together_with_comm_hooks, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardAllocFromPG::test_fully_shard_alloc_from_pg, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardForceSumReduction::test_fully_shard_force_sum_both_reductions, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardForceSumReduction::test_fully_shard_force_sum_reduce_scatter, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardReduceOpWorldSize1::test_size1_reduceop
2025-07-17T10:28:25.3419036Z 
2025-07-17T10:28:25.3419269Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:28:25.3419890Z Uploading artifacts took 0.00 seconds
2025-07-17T10:28:28.9905168Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:28:28.9906904Z   import pkg_resources
2025-07-17T10:28:28.9997713Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:28:28.9999171Z   import pkg_resources
2025-07-17T10:28:29.0208658Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:28:29.0210124Z   import pkg_resources
2025-07-17T10:28:29.0609915Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:28:29.0616509Z   import pkg_resources
2025-07-17T10:28:29.0708343Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:28:29.0709854Z   import pkg_resources
2025-07-17T10:28:29.0713611Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:28:29.0715078Z   import pkg_resources
2025-07-17T10:28:29.0763829Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:28:29.0765276Z   import pkg_resources
2025-07-17T10:28:29.0811020Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:28:29.0812479Z   import pkg_resources
2025-07-17T10:28:30.3411765Z Running test batch 'tests to run' cost 208.38 seconds
2025-07-17T10:28:30.9882166Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_multi_group --verbose
2025-07-17T10:28:34.0288747Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:28:34.0290929Z   import pkg_resources
2025-07-17T10:28:34.6921895Z No TD results found
2025-07-17T10:28:34.6923480Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:28:34.7175443Z Found test times from artifacts
2025-07-17T10:28:34.8203827Z Found test times from artifacts
2025-07-17T10:28:34.8244050Z Running all tests
2025-07-17T10:28:34.8248042Z Running parallel tests on 8 processes
2025-07-17T10:28:34.8248575Z Name: tests to run (est. time: 6.37min)
2025-07-17T10:28:34.8249123Z   Serial tests (1):
2025-07-17T10:28:34.8249833Z     distributed/_composable/fsdp/test_fully_shard_training 1/1
2025-07-17T10:28:34.8250351Z   Parallel tests (0):
2025-07-17T10:28:34.8250690Z Name: excluded (est. time: 0.0min)
2025-07-17T10:28:34.8251042Z   Serial tests (0):
2025-07-17T10:28:34.8251337Z   Parallel tests (0):
2025-07-17T10:28:34.8251932Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2025-07-17 10:28:34.824937]
2025-07-17T10:28:34.8252605Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:28:34.8255181Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_multi_group', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:28:34.825289]
2025-07-17T10:33:16.2929541Z 
2025-07-17T10:33:16.2931259Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_ddf60519671ff225_.log
2025-07-17T10:33:16.2934873Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_cpu_offload_eager, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_unshard_async_op
2025-07-17T10:33:16.2936924Z 
2025-07-17T10:33:16.2937213Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:33:16.2937733Z Uploading artifacts took 0.00 seconds
2025-07-17T10:33:20.0172470Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:33:20.0174322Z   import pkg_resources
2025-07-17T10:33:20.0571250Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:33:20.0573418Z   import pkg_resources
2025-07-17T10:33:20.0593836Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:33:20.0595275Z   import pkg_resources
2025-07-17T10:33:20.0800582Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:33:20.0802601Z   import pkg_resources
2025-07-17T10:33:20.0919919Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:33:20.0922156Z   import pkg_resources
2025-07-17T10:33:20.0958458Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:33:20.0959908Z   import pkg_resources
2025-07-17T10:33:20.0989017Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:33:20.0990461Z   import pkg_resources
2025-07-17T10:33:20.1009417Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:33:20.1010583Z   import pkg_resources
2025-07-17T10:33:21.4018761Z Running test batch 'tests to run' cost 286.58 seconds
2025-07-17T10:33:22.0533274Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_with_activation_checkpointing --verbose
2025-07-17T10:33:25.0770388Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:33:25.0772507Z   import pkg_resources
2025-07-17T10:33:25.7318760Z No TD results found
2025-07-17T10:33:25.7320421Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:33:25.7567823Z Found test times from artifacts
2025-07-17T10:33:25.8599049Z Found test times from artifacts
2025-07-17T10:33:25.8640391Z Running all tests
2025-07-17T10:33:25.8644500Z Running parallel tests on 8 processes
2025-07-17T10:33:25.8644882Z Name: tests to run (est. time: 6.37min)
2025-07-17T10:33:25.8645211Z   Serial tests (1):
2025-07-17T10:33:25.8645577Z     distributed/_composable/fsdp/test_fully_shard_training 1/1
2025-07-17T10:33:25.8645996Z   Parallel tests (0):
2025-07-17T10:33:25.8646279Z Name: excluded (est. time: 0.0min)
2025-07-17T10:33:25.8646577Z   Serial tests (0):
2025-07-17T10:33:25.8646822Z   Parallel tests (0):
2025-07-17T10:33:25.8647533Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2025-07-17 10:33:25.864581]
2025-07-17T10:33:25.8648147Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:33:25.8651939Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_with_activation_checkpointing', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:33:25.864944]
2025-07-17T10:34:27.2078642Z 
2025-07-17T10:34:27.2084130Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_b1285fe5540a5aee_.log
2025-07-17T10:34:27.2086373Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCompose::test_train_parity_with_activation_checkpointing
2025-07-17T10:34:27.2087210Z 
2025-07-17T10:34:27.2087437Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:34:27.2087846Z Uploading artifacts took 0.00 seconds
2025-07-17T10:34:30.7389479Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:34:30.7391017Z   import pkg_resources
2025-07-17T10:34:30.8916688Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:34:30.8918171Z   import pkg_resources
2025-07-17T10:34:30.9671007Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:34:30.9672493Z   import pkg_resources
2025-07-17T10:34:30.9687621Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:34:30.9689459Z   import pkg_resources
2025-07-17T10:34:30.9729432Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:34:30.9730862Z   import pkg_resources
2025-07-17T10:34:30.9806542Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:34:30.9807958Z   import pkg_resources
2025-07-17T10:34:30.9864330Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:34:30.9865763Z   import pkg_resources
2025-07-17T10:34:31.1345247Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:34:31.1346892Z   import pkg_resources
2025-07-17T10:34:32.5159722Z Running test batch 'tests to run' cost 66.65 seconds
2025-07-17T10:34:33.1624559Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_hsdp --verbose
2025-07-17T10:34:36.1749190Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:34:36.1751406Z   import pkg_resources
2025-07-17T10:34:36.8318916Z No TD results found
2025-07-17T10:34:36.8320226Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:34:36.8569952Z Found test times from artifacts
2025-07-17T10:34:36.9597796Z Found test times from artifacts
2025-07-17T10:34:36.9638895Z Running all tests
2025-07-17T10:34:36.9642551Z Running parallel tests on 8 processes
2025-07-17T10:34:36.9643097Z Name: tests to run (est. time: 6.37min)
2025-07-17T10:34:36.9643537Z   Serial tests (1):
2025-07-17T10:34:36.9644038Z     distributed/_composable/fsdp/test_fully_shard_training 1/1
2025-07-17T10:34:36.9644602Z   Parallel tests (0):
2025-07-17T10:34:36.9644982Z Name: excluded (est. time: 0.0min)
2025-07-17T10:34:36.9645397Z   Serial tests (0):
2025-07-17T10:34:36.9645739Z   Parallel tests (0):
2025-07-17T10:34:36.9646451Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2025-07-17 10:34:36.964368]
2025-07-17T10:34:36.9647280Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:34:36.9649582Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_hsdp', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:34:36.964690]
2025-07-17T10:35:00.8602709Z 
2025-07-17T10:35:00.8604265Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_2b20afd68a50618d_.log
2025-07-17T10:35:00.8606801Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDPTraining::test_train_parity_hsdp
2025-07-17T10:35:00.8607703Z 
2025-07-17T10:35:00.8608095Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:35:00.8608799Z Uploading artifacts took 0.00 seconds
2025-07-17T10:35:04.4854405Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:04.4856196Z   import pkg_resources
2025-07-17T10:35:04.5233972Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:04.5235531Z   import pkg_resources
2025-07-17T10:35:04.5793720Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:04.5795215Z   import pkg_resources
2025-07-17T10:35:04.6016722Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:04.6018372Z   import pkg_resources
2025-07-17T10:35:04.6189122Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:04.6190769Z   import pkg_resources
2025-07-17T10:35:04.6229878Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:04.6231310Z   import pkg_resources
2025-07-17T10:35:04.6386951Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:04.6388397Z   import pkg_resources
2025-07-17T10:35:04.6426527Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:04.6427725Z   import pkg_resources
2025-07-17T10:35:05.9193810Z Running test batch 'tests to run' cost 28.95 seconds
2025-07-17T10:35:06.5280380Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_2d_transformer_checkpoint_resume --verbose
2025-07-17T10:35:09.4968255Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:09.4971334Z   import pkg_resources
2025-07-17T10:35:10.1554828Z No TD results found
2025-07-17T10:35:10.1555977Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:35:10.1805940Z Found test times from artifacts
2025-07-17T10:35:10.2849840Z Found test times from artifacts
2025-07-17T10:35:10.2892545Z Running all tests
2025-07-17T10:35:10.2896395Z Running parallel tests on 8 processes
2025-07-17T10:35:10.2896923Z Name: tests to run (est. time: 6.37min)
2025-07-17T10:35:10.2897368Z   Serial tests (1):
2025-07-17T10:35:10.2897860Z     distributed/_composable/fsdp/test_fully_shard_training 1/1
2025-07-17T10:35:10.2898435Z   Parallel tests (0):
2025-07-17T10:35:10.2898823Z Name: excluded (est. time: 0.0min)
2025-07-17T10:35:10.2899236Z   Serial tests (0):
2025-07-17T10:35:10.2899577Z   Parallel tests (0):
2025-07-17T10:35:10.2900284Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2025-07-17 10:35:10.289750]
2025-07-17T10:35:10.2901098Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:35:10.2905403Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_2d_transformer_checkpoint_resume', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:35:10.290079]
2025-07-17T10:35:14.3599058Z 
2025-07-17T10:35:14.3601093Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_f6d783d6659b1b0c_.log
2025-07-17T10:35:14.3602820Z Running 0 items in this shard:
2025-07-17T10:35:14.3603099Z 
2025-07-17T10:35:14.3603474Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:35:14.3604145Z Uploading artifacts took 0.00 seconds
2025-07-17T10:35:18.0428332Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:18.0430519Z   import pkg_resources
2025-07-17T10:35:18.0454606Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:18.0456800Z   import pkg_resources
2025-07-17T10:35:18.0482615Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:18.0484575Z   import pkg_resources
2025-07-17T10:35:18.0866206Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:18.0868182Z   import pkg_resources
2025-07-17T10:35:18.0879458Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:18.0880928Z   import pkg_resources
2025-07-17T10:35:18.0954089Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:18.0956046Z   import pkg_resources
2025-07-17T10:35:18.1040421Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:18.1042389Z   import pkg_resources
2025-07-17T10:35:18.1170469Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:18.1172431Z   import pkg_resources
2025-07-17T10:35:19.7550665Z Running test batch 'tests to run' cost 9.47 seconds
2025-07-17T10:35:20.3708939Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_gradient_accumulation --verbose
2025-07-17T10:35:23.3631646Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:35:23.3634092Z   import pkg_resources
2025-07-17T10:35:24.0235878Z No TD results found
2025-07-17T10:35:24.0237082Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:35:24.0487409Z Found test times from artifacts
2025-07-17T10:35:24.1521677Z Found test times from artifacts
2025-07-17T10:35:24.1561567Z Running all tests
2025-07-17T10:35:24.1565281Z Running parallel tests on 8 processes
2025-07-17T10:35:24.1565907Z Name: tests to run (est. time: 6.37min)
2025-07-17T10:35:24.1566422Z   Serial tests (1):
2025-07-17T10:35:24.1566991Z     distributed/_composable/fsdp/test_fully_shard_training 1/1
2025-07-17T10:35:24.1567651Z   Parallel tests (0):
2025-07-17T10:35:24.1568117Z Name: excluded (est. time: 0.0min)
2025-07-17T10:35:24.1568539Z   Serial tests (0):
2025-07-17T10:35:24.1568835Z   Parallel tests (0):
2025-07-17T10:35:24.1569369Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2025-07-17 10:35:24.156619]
2025-07-17T10:35:24.1569943Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:35:24.1571369Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_gradient_accumulation', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:35:24.156939]
2025-07-17T10:36:17.7900468Z 
2025-07-17T10:36:17.7902282Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_555a340323e643ac_.log
2025-07-17T10:36:17.7904961Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_gradient_accumulation
2025-07-17T10:36:17.7906168Z 
2025-07-17T10:36:17.7906475Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:36:17.7907015Z Uploading artifacts took 0.00 seconds
2025-07-17T10:36:21.3868526Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:21.3869968Z   import pkg_resources
2025-07-17T10:36:21.4935519Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:21.4937161Z   import pkg_resources
2025-07-17T10:36:21.5159098Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:21.5160575Z   import pkg_resources
2025-07-17T10:36:21.5463214Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:21.5464634Z   import pkg_resources
2025-07-17T10:36:21.5691188Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:21.5692652Z   import pkg_resources
2025-07-17T10:36:21.5831094Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:21.5832711Z   import pkg_resources
2025-07-17T10:36:21.5867614Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:21.5869023Z   import pkg_resources
2025-07-17T10:36:21.5883832Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:21.5884974Z   import pkg_resources
2025-07-17T10:36:22.8403649Z Running test batch 'tests to run' cost 58.68 seconds
2025-07-17T10:36:23.4524751Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_state_dict.py -k test_dp_state_dict_save_load --verbose
2025-07-17T10:36:26.4373235Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:26.4375287Z   import pkg_resources
2025-07-17T10:36:27.0916633Z No TD results found
2025-07-17T10:36:27.0917988Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:36:27.1168135Z Found test times from artifacts
2025-07-17T10:36:27.2196412Z Found test times from artifacts
2025-07-17T10:36:27.2237588Z Running all tests
2025-07-17T10:36:27.2241321Z Running parallel tests on 8 processes
2025-07-17T10:36:27.2241869Z Name: tests to run (est. time: 0.18min)
2025-07-17T10:36:27.2242296Z   Serial tests (1):
2025-07-17T10:36:27.2242792Z     distributed/_composable/fsdp/test_fully_shard_state_dict 1/1
2025-07-17T10:36:27.2243368Z   Parallel tests (0):
2025-07-17T10:36:27.2243744Z Name: excluded (est. time: 0.0min)
2025-07-17T10:36:27.2244153Z   Serial tests (0):
2025-07-17T10:36:27.2244480Z   Parallel tests (0):
2025-07-17T10:36:27.2245188Z Running distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2025-07-17 10:36:27.224246]
2025-07-17T10:36:27.2245997Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:36:27.2248374Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_dp_state_dict_save_load', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:36:27.224572]
2025-07-17T10:36:42.8593423Z 
2025-07-17T10:36:42.8595073Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_42ad32f0f248fcc0_.log
2025-07-17T10:36:42.8601296Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_save_load
2025-07-17T10:36:42.8602058Z 
2025-07-17T10:36:42.8603048Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:36:42.8603771Z Uploading artifacts took 0.00 seconds
2025-07-17T10:36:46.4709141Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:46.4711189Z   import pkg_resources
2025-07-17T10:36:46.6096415Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:46.6098020Z   import pkg_resources
2025-07-17T10:36:46.6145519Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:46.6147002Z   import pkg_resources
2025-07-17T10:36:46.6227907Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:46.6229362Z   import pkg_resources
2025-07-17T10:36:46.6357205Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:46.6362998Z   import pkg_resources
2025-07-17T10:36:46.6444774Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:46.6446206Z   import pkg_resources
2025-07-17T10:36:46.6485398Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:46.6486841Z   import pkg_resources
2025-07-17T10:36:46.7191158Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:46.7192593Z   import pkg_resources
2025-07-17T10:36:48.1447225Z Running test batch 'tests to run' cost 20.92 seconds
2025-07-17T10:36:48.7612314Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_frozen.py --verbose
2025-07-17T10:36:51.7375077Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:36:51.7377357Z   import pkg_resources
2025-07-17T10:36:52.3911990Z No TD results found
2025-07-17T10:36:52.3913373Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:36:52.4167472Z Found test times from artifacts
2025-07-17T10:36:52.5200992Z Found test times from artifacts
2025-07-17T10:36:52.5243937Z Running all tests
2025-07-17T10:36:52.5247674Z Running parallel tests on 8 processes
2025-07-17T10:36:52.5248200Z Name: tests to run (est. time: 0.64min)
2025-07-17T10:36:52.5248651Z   Serial tests (1):
2025-07-17T10:36:52.5249492Z     distributed/_composable/fsdp/test_fully_shard_frozen 1/1
2025-07-17T10:36:52.5249966Z   Parallel tests (0):
2025-07-17T10:36:52.5250197Z Name: excluded (est. time: 0.0min)
2025-07-17T10:36:52.5250445Z   Serial tests (0):
2025-07-17T10:36:52.5250649Z   Parallel tests (0):
2025-07-17T10:36:52.5251067Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2025-07-17 10:36:52.524876]
2025-07-17T10:36:52.5251527Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:36:52.5253785Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:36:52.525199]
2025-07-17T10:37:47.0096819Z 
2025-07-17T10:37:47.0098308Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_567746345b443db2_.log
2025-07-17T10:37:47.0102133Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group
2025-07-17T10:37:47.0104976Z 
2025-07-17T10:37:47.0105691Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:37:47.0106118Z Uploading artifacts took 0.00 seconds
2025-07-17T10:37:50.6217045Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:37:50.6219295Z   import pkg_resources
2025-07-17T10:37:50.6258336Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:37:50.6260299Z   import pkg_resources
2025-07-17T10:37:50.7049770Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:37:50.7051866Z   import pkg_resources
2025-07-17T10:37:50.7102643Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:37:50.7104169Z   import pkg_resources
2025-07-17T10:37:50.7220950Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:37:50.7222773Z   import pkg_resources
2025-07-17T10:37:50.7253537Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:37:50.7255133Z   import pkg_resources
2025-07-17T10:37:50.7306488Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:37:50.7308490Z   import pkg_resources
2025-07-17T10:37:50.7311629Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:37:50.7313222Z   import pkg_resources
2025-07-17T10:37:52.0592295Z Running test batch 'tests to run' cost 59.53 seconds
2025-07-17T10:37:52.6780430Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_mixed_precision.py -k test_compute_dtype --verbose
2025-07-17T10:37:55.6686143Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:37:55.6688239Z   import pkg_resources
2025-07-17T10:37:56.3232763Z No TD results found
2025-07-17T10:37:56.3233577Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:37:56.3481563Z Found test times from artifacts
2025-07-17T10:37:56.4516101Z Found test times from artifacts
2025-07-17T10:37:56.4557122Z Running all tests
2025-07-17T10:37:56.4561012Z Running parallel tests on 8 processes
2025-07-17T10:37:56.4561636Z Name: tests to run (est. time: 0.18min)
2025-07-17T10:37:56.4562174Z   Serial tests (1):
2025-07-17T10:37:56.4562817Z     distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1
2025-07-17T10:37:56.4563562Z   Parallel tests (0):
2025-07-17T10:37:56.4563997Z Name: excluded (est. time: 0.0min)
2025-07-17T10:37:56.4564361Z   Serial tests (0):
2025-07-17T10:37:56.4564625Z   Parallel tests (0):
2025-07-17T10:37:56.4565176Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2025-07-17 10:37:56.456197]
2025-07-17T10:37:56.4565785Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:37:56.4567180Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_compute_dtype', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:37:56.456512]
2025-07-17T10:38:06.6842745Z 
2025-07-17T10:38:06.6844623Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_e2c30a6c271e681e_.log
2025-07-17T10:38:06.6847445Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype
2025-07-17T10:38:06.6848566Z 
2025-07-17T10:38:06.6849464Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:38:06.6850210Z Uploading artifacts took 0.00 seconds
2025-07-17T10:38:10.3701647Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:10.3704058Z   import pkg_resources
2025-07-17T10:38:10.4583856Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:10.4585932Z   import pkg_resources
2025-07-17T10:38:10.4598457Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:10.4599723Z   import pkg_resources
2025-07-17T10:38:10.4691224Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:10.4693225Z   import pkg_resources
2025-07-17T10:38:10.4793946Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:10.4796296Z   import pkg_resources
2025-07-17T10:38:10.4964363Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:10.4966449Z   import pkg_resources
2025-07-17T10:38:10.4989995Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:10.4991427Z   import pkg_resources
2025-07-17T10:38:10.5034344Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:10.5036335Z   import pkg_resources
2025-07-17T10:38:11.8249966Z Running test batch 'tests to run' cost 15.37 seconds
2025-07-17T10:38:12.4384149Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_mixed_precision.py -k test_reduce_dtype --verbose
2025-07-17T10:38:15.4164787Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:15.4166930Z   import pkg_resources
2025-07-17T10:38:16.0710611Z No TD results found
2025-07-17T10:38:16.0711520Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:38:16.0961439Z Found test times from artifacts
2025-07-17T10:38:16.1993091Z Found test times from artifacts
2025-07-17T10:38:16.2034766Z Running all tests
2025-07-17T10:38:16.2038414Z Running parallel tests on 8 processes
2025-07-17T10:38:16.2038952Z Name: tests to run (est. time: 0.18min)
2025-07-17T10:38:16.2039409Z   Serial tests (1):
2025-07-17T10:38:16.2040233Z     distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1
2025-07-17T10:38:16.2040850Z   Parallel tests (0):
2025-07-17T10:38:16.2041235Z Name: excluded (est. time: 0.0min)
2025-07-17T10:38:16.2041660Z   Serial tests (0):
2025-07-17T10:38:16.2042015Z   Parallel tests (0):
2025-07-17T10:38:16.2042763Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2025-07-17 10:38:16.203939]
2025-07-17T10:38:16.2043619Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:38:16.2045828Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_reduce_dtype', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:38:16.204269]
2025-07-17T10:38:26.4325372Z 
2025-07-17T10:38:26.4327051Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_5533805475a6d986_.log
2025-07-17T10:38:26.4329916Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype
2025-07-17T10:38:26.4331134Z 
2025-07-17T10:38:26.4331590Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:38:26.4332290Z Uploading artifacts took 0.00 seconds
2025-07-17T10:38:30.1032311Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:30.1033870Z   import pkg_resources
2025-07-17T10:38:30.1937845Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:30.1939335Z   import pkg_resources
2025-07-17T10:38:30.2287370Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:30.2288832Z   import pkg_resources
2025-07-17T10:38:30.2356455Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:30.2357959Z   import pkg_resources
2025-07-17T10:38:30.2396731Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:30.2398589Z   import pkg_resources
2025-07-17T10:38:30.2449767Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:30.2451465Z   import pkg_resources
2025-07-17T10:38:30.2470105Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:30.2472464Z   import pkg_resources
2025-07-17T10:38:30.2668831Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:30.2670263Z   import pkg_resources
2025-07-17T10:38:31.7804541Z Running test batch 'tests to run' cost 15.58 seconds
2025-07-17T10:38:32.3933556Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py -k test_clip_grad_norm_2d --verbose
2025-07-17T10:38:35.3909245Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:35.3911753Z   import pkg_resources
2025-07-17T10:38:36.0481076Z No TD results found
2025-07-17T10:38:36.0482891Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:38:36.0732315Z Found test times from artifacts
2025-07-17T10:38:36.1768721Z Found test times from artifacts
2025-07-17T10:38:36.1809625Z Running all tests
2025-07-17T10:38:36.1813578Z Running parallel tests on 8 processes
2025-07-17T10:38:36.1814182Z Name: tests to run (est. time: 0.0min)
2025-07-17T10:38:36.1814690Z   Serial tests (1):
2025-07-17T10:38:36.1815304Z     distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1
2025-07-17T10:38:36.1816174Z   Parallel tests (0):
2025-07-17T10:38:36.1816622Z Name: excluded (est. time: 0.0min)
2025-07-17T10:38:36.1816909Z   Serial tests (0):
2025-07-17T10:38:36.1817143Z   Parallel tests (0):
2025-07-17T10:38:36.1817587Z Running distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 ... [2025-07-17 10:38:36.181455]
2025-07-17T10:38:36.1818074Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:38:36.1819583Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_clip_grad_norm_2d', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:38:36.181787]
2025-07-17T10:38:40.4521477Z 
2025-07-17T10:38:40.4523036Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_93c132972ce89d32_.log
2025-07-17T10:38:40.4525800Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize4::test_clip_grad_norm_2d
2025-07-17T10:38:40.4526966Z 
2025-07-17T10:38:40.4527454Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:38:40.4528094Z Uploading artifacts took 0.00 seconds
2025-07-17T10:38:44.1583489Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:44.1585291Z   import pkg_resources
2025-07-17T10:38:44.2114619Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:44.2116309Z   import pkg_resources
2025-07-17T10:38:44.2334202Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:44.2335641Z   import pkg_resources
2025-07-17T10:38:44.2419821Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:44.2421301Z   import pkg_resources
2025-07-17T10:38:44.2426059Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:44.2427272Z   import pkg_resources
2025-07-17T10:38:44.2463705Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:44.2465376Z   import pkg_resources
2025-07-17T10:38:44.2502854Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:44.2504313Z   import pkg_resources
2025-07-17T10:38:44.3474635Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:44.3476621Z   import pkg_resources
2025-07-17T10:38:45.7971275Z Running test batch 'tests to run' cost 9.62 seconds
2025-07-17T10:38:46.4111490Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_compile.py --verbose
2025-07-17T10:38:49.4014990Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:38:49.4017274Z   import pkg_resources
2025-07-17T10:38:50.0549615Z No TD results found
2025-07-17T10:38:50.0550789Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:38:50.0803825Z Found test times from artifacts
2025-07-17T10:38:50.1838628Z Found test times from artifacts
2025-07-17T10:38:50.1879153Z Running all tests
2025-07-17T10:38:50.1883247Z Running parallel tests on 8 processes
2025-07-17T10:38:50.1883775Z Name: tests to run (est. time: 3.83min)
2025-07-17T10:38:50.1884207Z   Serial tests (1):
2025-07-17T10:38:50.1884694Z     distributed/_composable/fsdp/test_fully_shard_compile 1/1
2025-07-17T10:38:50.1885255Z   Parallel tests (0):
2025-07-17T10:38:50.1885638Z Name: excluded (est. time: 0.0min)
2025-07-17T10:38:50.1886268Z   Serial tests (0):
2025-07-17T10:38:50.1886613Z   Parallel tests (0):
2025-07-17T10:38:50.1887312Z Running distributed/_composable/fsdp/test_fully_shard_compile 1/1 ... [2025-07-17 10:38:50.188423]
2025-07-17T10:38:50.1888100Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:38:50.1889545Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:38:50.188747]
2025-07-17T10:44:32.3421736Z 
2025-07-17T10:44:32.3423536Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_5548ca3a5e7cdf5d_.log
2025-07-17T10:44:32.3434396Z Running 18 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompileCompute::test_disable_compiling_hooks, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_compiled_autograd_ctx, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_recompiles_on_fsdp_layers, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_trace_use_training_state, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_True, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_True_graph_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_inductor, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_trace_fsdp_copy_, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True_graph_partition
2025-07-17T10:44:32.3443289Z 
2025-07-17T10:44:32.3443518Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:44:32.3444053Z Uploading artifacts took 0.00 seconds
2025-07-17T10:44:35.8218709Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:44:35.8220838Z   import pkg_resources
2025-07-17T10:44:36.0426201Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:44:36.0428615Z   import pkg_resources
2025-07-17T10:44:36.1119786Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:44:36.1122014Z   import pkg_resources
2025-07-17T10:44:36.1124645Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:44:36.1125831Z   import pkg_resources
2025-07-17T10:44:36.1195843Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:44:36.1197876Z   import pkg_resources
2025-07-17T10:44:36.1239970Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:44:36.1242406Z   import pkg_resources
2025-07-17T10:44:36.1282484Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:44:36.1284450Z   import pkg_resources
2025-07-17T10:44:36.1514106Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:44:36.1516065Z   import pkg_resources
2025-07-17T10:44:37.7595352Z Running test batch 'tests to run' cost 347.57 seconds
2025-07-17T10:44:38.3789250Z + python test/run_test.py -i distributed/fsdp/test_fsdp_tp_integration.py -k test_fsdp_tp_integration --verbose
2025-07-17T10:44:41.3604103Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:44:41.3606213Z   import pkg_resources
2025-07-17T10:44:42.0153590Z No TD results found
2025-07-17T10:44:42.0155535Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:44:42.0401779Z Found test times from artifacts
2025-07-17T10:44:42.1443898Z Found test times from artifacts
2025-07-17T10:44:42.1485300Z Running all tests
2025-07-17T10:44:42.1492455Z Running parallel tests on 8 processes
2025-07-17T10:44:42.1492978Z Name: tests to run (est. time: 0.49min)
2025-07-17T10:44:42.1493423Z   Serial tests (1):
2025-07-17T10:44:42.1493854Z     distributed/fsdp/test_fsdp_tp_integration 1/1
2025-07-17T10:44:42.1494363Z   Parallel tests (0):
2025-07-17T10:44:42.1495020Z Name: excluded (est. time: 0.0min)
2025-07-17T10:44:42.1495441Z   Serial tests (0):
2025-07-17T10:44:42.1495795Z   Parallel tests (0):
2025-07-17T10:44:42.1496646Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2025-07-17 10:44:42.149366]
2025-07-17T10:44:42.1497662Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:44:42.1499333Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_fsdp_tp_integration', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:44:42.149703]
2025-07-17T10:45:30.2250015Z 
2025-07-17T10:45:30.2251454Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_99cb5f40693750a9_.log
2025-07-17T10:45:30.2257871Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_extension_grad, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_sync_module_state
2025-07-17T10:45:30.2261731Z 
2025-07-17T10:45:30.2262002Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:45:30.2262505Z Uploading artifacts took 0.00 seconds
2025-07-17T10:45:33.8129416Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:45:33.8132150Z   import pkg_resources
2025-07-17T10:45:33.8928879Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:45:33.8930906Z   import pkg_resources
2025-07-17T10:45:33.8972404Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:45:33.8974373Z   import pkg_resources
2025-07-17T10:45:33.9290009Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:45:33.9292026Z   import pkg_resources
2025-07-17T10:45:33.9638052Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:45:33.9639516Z   import pkg_resources
2025-07-17T10:45:33.9708401Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:45:33.9710400Z   import pkg_resources
2025-07-17T10:45:33.9739897Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:45:33.9742132Z   import pkg_resources
2025-07-17T10:45:33.9940184Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:45:33.9942151Z   import pkg_resources
2025-07-17T10:45:35.2366652Z Running test batch 'tests to run' cost 53.09 seconds
2025-07-17T10:45:35.8493152Z + python test/run_test.py --include distributed/test_dynamo_distributed distributed/test_inductor_collectives distributed/test_compute_comm_reordering --verbose
2025-07-17T10:45:38.8230002Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T10:45:38.8231858Z   import pkg_resources
2025-07-17T10:45:39.4753582Z No TD results found
2025-07-17T10:45:39.4754959Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-17T10:45:39.5007416Z Found test times from artifacts
2025-07-17T10:45:39.6047208Z Found test times from artifacts
2025-07-17T10:45:39.6088770Z Running all tests
2025-07-17T10:45:39.6092823Z Running parallel tests on 8 processes
2025-07-17T10:45:39.6093374Z Name: tests to run (est. time: 19.79min)
2025-07-17T10:45:39.6093830Z   Serial tests (4):
2025-07-17T10:45:39.6094260Z     distributed/test_compute_comm_reordering 1/1
2025-07-17T10:45:39.6094819Z     distributed/test_dynamo_distributed 1/2
2025-07-17T10:45:39.6095322Z     distributed/test_dynamo_distributed 2/2
2025-07-17T10:45:39.6095837Z     distributed/test_inductor_collectives 1/1
2025-07-17T10:45:39.6096340Z   Parallel tests (0):
2025-07-17T10:45:39.6096578Z Name: excluded (est. time: 0.0min)
2025-07-17T10:45:39.6096837Z   Serial tests (0):
2025-07-17T10:45:39.6097044Z   Parallel tests (0):
2025-07-17T10:45:39.6097420Z Running distributed/test_compute_comm_reordering 1/1 ... [2025-07-17 10:45:39.609436]
2025-07-17T10:45:39.6097854Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:45:39.6099451Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:45:39.609770]
2025-07-17T10:47:26.5628556Z 
2025-07-17T10:47:26.5629646Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_0ebed5d6e342051e_.log
2025-07-17T10:47:26.5635453Z Running 8 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_inductor_default_comms_ordering, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms
2025-07-17T10:47:26.5639848Z 
2025-07-17T10:47:26.5640130Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T10:47:26.5640639Z Uploading artifacts took 0.00 seconds
2025-07-17T10:47:26.5641134Z Running distributed/test_dynamo_distributed 1/2 ... [2025-07-17 10:47:26.562911]
2025-07-17T10:47:26.5641650Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:47:26.5642630Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:47:26.563233]
2025-07-17T10:57:17.4642452Z 
2025-07-17T10:57:17.4644036Z distributed/test_dynamo_distributed 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.2_de4d32cd8948f688_.log
2025-07-17T10:57:17.4673235Z Running 37 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_ddp_optimizer_inductor_strides_dont_specialize, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_no_binding, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_scalar, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_speculation_divergence, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_tensor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_dim_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_scalar_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_type_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_no_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_get_pg_attr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_guard_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_full_model_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split
2025-07-17T10:57:17.4685778Z 
2025-07-17T10:57:17.4686001Z Running distributed/test_dynamo_distributed 2/2 ... [2025-07-17 10:57:17.464483]
2025-07-17T10:57:17.4686426Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T10:57:17.4687357Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 10:57:17.464840]
2025-07-17T11:07:25.3516087Z 
2025-07-17T11:07:25.3517176Z PRINTING LOG FILE of distributed/test_dynamo_distributed 2/2 (test/test-reports/distributed.test_dynamo_distributed_2.2_492ff260544723ef_.log)
2025-07-17T11:07:25.3519920Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3521934Z   import pkg_resources
2025-07-17T11:07:25.3522692Z Test results will be stored in test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-2c4842f0210cd3ab.xml
2025-07-17T11:07:25.3523577Z ============================= test session starts ==============================
2025-07-17T11:07:25.3524211Z platform linux -- Python 3.10.18, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python
2025-07-17T11:07:25.3524758Z cachedir: .pytest_cache
2025-07-17T11:07:25.3525278Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-07-17T11:07:25.3525827Z rootdir: /var/lib/jenkins/pytorch
2025-07-17T11:07:25.3526105Z configfile: pytest.ini
2025-07-17T11:07:25.3526634Z plugins: xdist-3.3.1, rerunfailures-14.0, xdoctest-1.1.0, flakefinder-1.1.0, cpp-2.3.0, subtests-0.13.1, hypothesis-5.35.1, typeguard-4.3.0
2025-07-17T11:07:25.3527192Z collecting ... collected 60 items
2025-07-17T11:07:25.3527491Z stepcurrent: Cannot find last run test, not skipping
2025-07-17T11:07:25.3535345Z Running 23 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_local_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training
2025-07-17T11:07:25.3543355Z 
2025-07-17T11:07:25.3543764Z distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting PASSED [2.1104s] [  4%]
2025-07-17T11:07:25.3544707Z distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct PASSED [1.5530s] [  8%]
2025-07-17T11:07:25.3545696Z distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect PASSED [2.3988s] [ 13%]
2025-07-17T11:07:25.3546703Z distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi PASSED [1.0515s] [ 17%]
2025-07-17T11:07:25.3548012Z distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation I0717 10:57:30.968000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1750901
2025-07-17T11:07:25.3549149Z I0717 10:57:30.969000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1750902
2025-07-17T11:07:25.3550065Z I0717 10:57:30.970000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1750903
2025-07-17T11:07:25.3550971Z I0717 10:57:30.971000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1750904
2025-07-17T11:07:25.3551819Z I0717 10:57:30.972000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1750905
2025-07-17T11:07:25.3552657Z I0717 10:57:30.973000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1750906
2025-07-17T11:07:25.3553501Z I0717 10:57:30.974000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1750907
2025-07-17T11:07:25.3554309Z I0717 10:57:30.975000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1750908
2025-07-17T11:07:25.3555762Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3556983Z   import pkg_resources
2025-07-17T11:07:25.3558408Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3559597Z   import pkg_resources
2025-07-17T11:07:25.3560722Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3561987Z   import pkg_resources
2025-07-17T11:07:25.3563199Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3564370Z   import pkg_resources
2025-07-17T11:07:25.3565631Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3566804Z   import pkg_resources
2025-07-17T11:07:25.3567950Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3569114Z   import pkg_resources
2025-07-17T11:07:25.3570240Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3571580Z   import pkg_resources
2025-07-17T11:07:25.3572765Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3573924Z   import pkg_resources
2025-07-17T11:07:25.3574151Z PASSED [28.0507s] [ 21%]
2025-07-17T11:07:25.3575068Z distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache I0717 10:57:59.019000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1755207
2025-07-17T11:07:25.3576321Z I0717 10:57:59.020000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1755208
2025-07-17T11:07:25.3577144Z I0717 10:57:59.021000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1755209
2025-07-17T11:07:25.3577982Z I0717 10:57:59.022000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1755210
2025-07-17T11:07:25.3578835Z I0717 10:57:59.023000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1755211
2025-07-17T11:07:25.3579657Z I0717 10:57:59.024000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1755212
2025-07-17T11:07:25.3580465Z I0717 10:57:59.025000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1755213
2025-07-17T11:07:25.3581370Z I0717 10:57:59.026000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1755214
2025-07-17T11:07:25.3582899Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3584074Z   import pkg_resources
2025-07-17T11:07:25.3585304Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3586463Z   import pkg_resources
2025-07-17T11:07:25.3587617Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3588767Z   import pkg_resources
2025-07-17T11:07:25.3589933Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3591114Z   import pkg_resources
2025-07-17T11:07:25.3592220Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3593370Z   import pkg_resources
2025-07-17T11:07:25.3594586Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3595756Z   import pkg_resources
2025-07-17T11:07:25.3596866Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3598033Z   import pkg_resources
2025-07-17T11:07:25.3599179Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3600362Z   import pkg_resources
2025-07-17T11:07:25.3600893Z [rank2]:W0717 10:58:08.553000 1755209 site-packages/torch/_inductor/compile_fx.py:1168] [0/0] Sleeping for 10 since sleep_sec_TESTING_ONLY is set
2025-07-17T11:07:25.3601804Z [rank1]:W0717 10:58:10.373000 1755208 site-packages/torch/_inductor/compile_fx.py:1168] [0/0] Sleeping for 10 since sleep_sec_TESTING_ONLY is set
2025-07-17T11:07:25.3602666Z [rank6]:W0717 10:58:12.245000 1755213 site-packages/torch/_inductor/compile_fx.py:1168] [0/0] Sleeping for 10 since sleep_sec_TESTING_ONLY is set
2025-07-17T11:07:25.3603514Z [rank5]:W0717 10:58:14.054000 1755212 site-packages/torch/_inductor/compile_fx.py:1168] [0/0] Sleeping for 10 since sleep_sec_TESTING_ONLY is set
2025-07-17T11:07:25.3604440Z [rank7]:W0717 10:58:15.977000 1755214 site-packages/torch/_inductor/compile_fx.py:1168] [0/0] Sleeping for 10 since sleep_sec_TESTING_ONLY is set
2025-07-17T11:07:25.3605356Z [rank3]:W0717 10:58:17.855000 1755210 site-packages/torch/_inductor/compile_fx.py:1168] [0/0] Sleeping for 10 since sleep_sec_TESTING_ONLY is set
2025-07-17T11:07:25.3606233Z [rank0]:W0717 10:58:19.795000 1755207 site-packages/torch/_inductor/compile_fx.py:1168] [0/0] Sleeping for 10 since sleep_sec_TESTING_ONLY is set
2025-07-17T11:07:25.3607072Z [rank4]:W0717 10:58:19.814000 1755211 site-packages/torch/_inductor/compile_fx.py:1168] [0/0] Sleeping for 10 since sleep_sec_TESTING_ONLY is set
2025-07-17T11:07:25.3607995Z [rank0]:W0717 10:58:32.532000 1755207 site-packages/torch/_inductor/compile_fx.py:1168] [0/0] Sleeping for 10 since sleep_sec_TESTING_ONLY is set
2025-07-17T11:07:25.3608555Z PASSED [46.4766s] [ 26%]
2025-07-17T11:07:25.3609496Z distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective I0717 10:58:45.497000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1763331
2025-07-17T11:07:25.3610762Z I0717 10:58:45.499000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1763332
2025-07-17T11:07:25.3611617Z I0717 10:58:45.500000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1763333
2025-07-17T11:07:25.3612466Z I0717 10:58:45.501000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1763334
2025-07-17T11:07:25.3613288Z I0717 10:58:45.502000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1763335
2025-07-17T11:07:25.3614094Z I0717 10:58:45.503000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1763336
2025-07-17T11:07:25.3614921Z I0717 10:58:45.504000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1763337
2025-07-17T11:07:25.3615820Z I0717 10:58:45.505000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1763338
2025-07-17T11:07:25.3617392Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3618574Z   import pkg_resources
2025-07-17T11:07:25.3619697Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3620864Z   import pkg_resources
2025-07-17T11:07:25.3621990Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3623134Z   import pkg_resources
2025-07-17T11:07:25.3624268Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3625438Z   import pkg_resources
2025-07-17T11:07:25.3626733Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3627913Z   import pkg_resources
2025-07-17T11:07:25.3629036Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3630270Z   import pkg_resources
2025-07-17T11:07:25.3631377Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3632532Z   import pkg_resources
2025-07-17T11:07:25.3633636Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3634795Z   import pkg_resources
2025-07-17T11:07:25.3635003Z woof
2025-07-17T11:07:25.3635190Z woof
2025-07-17T11:07:25.3635367Z woof
2025-07-17T11:07:25.3635531Z woof
2025-07-17T11:07:25.3635727Z woof
2025-07-17T11:07:25.3635892Z woof
2025-07-17T11:07:25.3636057Z woof
2025-07-17T11:07:25.3636212Z woof
2025-07-17T11:07:25.3636372Z woof
2025-07-17T11:07:25.3636537Z woof
2025-07-17T11:07:25.3636703Z woof
2025-07-17T11:07:25.3636866Z woof
2025-07-17T11:07:25.3637036Z woof
2025-07-17T11:07:25.3637200Z woof
2025-07-17T11:07:25.3637363Z woof
2025-07-17T11:07:25.3637545Z woof
2025-07-17T11:07:25.3637707Z woof
2025-07-17T11:07:25.3637886Z woof
2025-07-17T11:07:25.3638065Z woof
2025-07-17T11:07:25.3638244Z woof
2025-07-17T11:07:25.3638518Z woof
2025-07-17T11:07:25.3638683Z woof
2025-07-17T11:07:25.3638837Z woof
2025-07-17T11:07:25.3638997Z woof
2025-07-17T11:07:25.3639194Z PASSED [27.9499s] [ 30%]
2025-07-17T11:07:25.3640071Z distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_missing_source I0717 10:59:13.449000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1769758
2025-07-17T11:07:25.3641225Z I0717 10:59:13.450000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1769759
2025-07-17T11:07:25.3642126Z I0717 10:59:13.451000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1769760
2025-07-17T11:07:25.3642938Z I0717 10:59:13.452000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1769761
2025-07-17T11:07:25.3643756Z I0717 10:59:13.453000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1769762
2025-07-17T11:07:25.3644563Z I0717 10:59:13.454000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1769763
2025-07-17T11:07:25.3645372Z I0717 10:59:13.455000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1769764
2025-07-17T11:07:25.3646181Z I0717 10:59:13.456000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1769765
2025-07-17T11:07:25.3647623Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3648781Z   import pkg_resources
2025-07-17T11:07:25.3650045Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3651215Z   import pkg_resources
2025-07-17T11:07:25.3652322Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3653559Z   import pkg_resources
2025-07-17T11:07:25.3654656Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3655813Z   import pkg_resources
2025-07-17T11:07:25.3656999Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3658196Z   import pkg_resources
2025-07-17T11:07:25.3659298Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3660455Z   import pkg_resources
2025-07-17T11:07:25.3661579Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3662832Z   import pkg_resources
2025-07-17T11:07:25.3663949Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3665182Z   import pkg_resources
2025-07-17T11:07:25.3665427Z PASSED [27.9500s] [ 34%]
2025-07-17T11:07:25.3666234Z distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph I0717 10:59:41.401000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1777265
2025-07-17T11:07:25.3667345Z I0717 10:59:41.402000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1777266
2025-07-17T11:07:25.3668156Z I0717 10:59:41.403000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1777267
2025-07-17T11:07:25.3669001Z I0717 10:59:41.404000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1777268
2025-07-17T11:07:25.3669832Z I0717 10:59:41.405000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1777269
2025-07-17T11:07:25.3670634Z I0717 10:59:41.406000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1777270
2025-07-17T11:07:25.3671441Z I0717 10:59:41.407000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1777271
2025-07-17T11:07:25.3672384Z I0717 10:59:41.408000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1777272
2025-07-17T11:07:25.3673897Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3675159Z   import pkg_resources
2025-07-17T11:07:25.3676290Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3677447Z   import pkg_resources
2025-07-17T11:07:25.3678553Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3679717Z   import pkg_resources
2025-07-17T11:07:25.3680876Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3682105Z   import pkg_resources
2025-07-17T11:07:25.3683204Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3684439Z   import pkg_resources
2025-07-17T11:07:25.3685546Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3686694Z   import pkg_resources
2025-07-17T11:07:25.3687813Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3688985Z   import pkg_resources
2025-07-17T11:07:25.3690095Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.3691307Z   import pkg_resources
2025-07-17T11:07:25.3691945Z [rank7]:W0717 11:00:03.156000 1777272 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.3693024Z [rank1]:W0717 11:00:03.156000 1777266 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.3694076Z [rank0]:W0717 11:00:03.156000 1777265 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.3695215Z [rank6]:W0717 11:00:03.159000 1777271 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.3696461Z [rank5]:W0717 11:00:03.159000 1777270 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.3697559Z [rank2]:W0717 11:00:03.159000 1777267 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.3698723Z [rank3]:W0717 11:00:03.160000 1777268 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.3699780Z [rank4]:W0717 11:00:03.162000 1777269 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.3700666Z Memory access fault by GPU node-7 (Agent handle: 0x55eb10de45d0) on address 0x7fde66823000. Reason: Unknown.
2025-07-17T11:07:25.3701351Z Memory access fault by GPU node-6 (Agent handle: 0x562badd93d60) on address 0x7faea3403000. Reason: Unknown.
2025-07-17T11:07:25.3702050Z Memory access fault by GPU node-5 (Agent handle: 0x55e6a3e43e50) on address 0x7fba7be06000. Reason: Unknown.
2025-07-17T11:07:25.3702753Z Memory access fault by GPU node-4 (Agent handle: 0x55d3b8a4f5d0) on address 0x7f7eeb408000. Reason: Unknown.
2025-07-17T11:07:25.3703421Z Memory access fault by GPU node-8 (Agent handle: 0x56166ebb8e70) on address 0x7f7967e04000. Reason: Unknown.
2025-07-17T11:07:25.3704093Z Memory access fault by GPU node-2 (Agent handle: 0x55873a30bc10) on address 0x7f81eaa23000. Reason: Unknown.
2025-07-17T11:07:25.3704742Z Memory access fault by GPU node-9 (Agent handle: 0x561986fe72b0) on address 0x7f0f4de07000. Reason: Unknown.
2025-07-17T11:07:25.3705452Z Memory access fault by GPU node-3 (Agent handle: 0x55b7510c4b90) on address 0x7f996ea1d000. Reason: Unknown.
2025-07-17T11:07:25.3705931Z SIGABRT(6), PID: 1777269, Thread 1777269: 
2025-07-17T11:07:25.3706746Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3707577Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3708135Z frame #2: pthread_kill + 0x12c (0x7fb86309a9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3708623Z frame #3: raise + 0x16 (0x7fb863046476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3709284Z frame #4: <unknown function> + 0x4feaa4d (0x7fb6bfc22a4d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/_C/libtriton.so)
2025-07-17T11:07:25.3710007Z frame #5: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3710576Z frame #6: <unknown function> + 0x77435 (0x7fb6df2f8435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.3711165Z frame #7: <unknown function> + 0x76c8e (0x7fb6df2f7c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.3711756Z frame #8: <unknown function> + 0x6ad69 (0x7fb6df2ebd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.3712346Z frame #9: <unknown function> + 0x3f54db (0x7fb76ff504db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3712930Z frame #10: <unknown function> + 0x3e11d8 (0x7fb76ff3c1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3713495Z frame #11: <unknown function> + 0xb2f86 (0x7fb76fc0df86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3714037Z frame #12: <unknown function> + 0x8e501 (0x7fb76fbe9501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3714777Z frame #13: c10::hip::device_synchronize() + 0x5a (0x7fb79a9f07ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.3715782Z frame #14: <unknown function> + 0xd307d9 (0x7fb7b8f3a7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.3716597Z frame #15: <unknown function> + 0x139902 (0x562ba4742902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3717216Z frame #16: _PyEval_EvalFrameDefault + 0x49b5 (0x562ba4739a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3717824Z frame #17: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3718436Z frame #18: _PyEval_EvalFrameDefault + 0x49b5 (0x562ba4739a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3719103Z frame #19: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3719689Z frame #20: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3720288Z frame #21: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3720905Z frame #22: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3721547Z frame #23: <unknown function> + 0x1474b2 (0x562ba47504b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3722100Z frame #24: PyObject_Call + 0xbc (0x562ba4750e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3722656Z frame #25: <unknown function> + 0x1ff69d (0x562ba480869d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3723259Z frame #26: _PyObject_MakeTpCall + 0x2d3 (0x562ba473e023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3723864Z frame #27: _PyEval_EvalFrameDefault + 0x4c16 (0x562ba4739cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3724520Z frame #28: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3725124Z frame #29: _PyEval_EvalFrameDefault + 0x30c (0x562ba47353bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3725722Z frame #30: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3726307Z frame #31: _PyEval_EvalFrameDefault + 0x30c (0x562ba47353bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3727010Z frame #32: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3727609Z frame #33: _PyEval_EvalFrameDefault + 0x49b5 (0x562ba4739a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3728227Z frame #34: _PyObject_FastCallDictTstate + 0xd0 (0x562ba473d4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3728832Z frame #35: _PyObject_Call_Prepend + 0x69 (0x562ba474eab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3729393Z frame #36: <unknown function> + 0x205949 (0x562ba480e949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3729959Z frame #37: _PyObject_MakeTpCall + 0x2d3 (0x562ba473e023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3730563Z frame #38: _PyEval_EvalFrameDefault + 0x4c16 (0x562ba4739cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3731174Z frame #39: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3731776Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x562ba47353bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3732442Z frame #41: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3733035Z frame #42: _PyEval_EvalFrameDefault + 0x30c (0x562ba47353bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3733631Z frame #43: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3734205Z frame #44: _PyEval_EvalFrameDefault + 0x30c (0x562ba47353bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3734800Z frame #45: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3735420Z frame #46: _PyEval_EvalFrameDefault + 0x1340 (0x562ba47363f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3736191Z frame #47: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3737014Z frame #48: <unknown function> + 0x7ab3f7 (0x7fb7b89b53f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.3737748Z frame #49: <unknown function> + 0x13bc88 (0x562ba4744c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3738323Z frame #50: PyObject_Call + 0x20f (0x562ba4750f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3738974Z frame #51: _PyEval_EvalFrameDefault + 0x58dd (0x562ba473a98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3739572Z frame #52: <unknown function> + 0x147608 (0x562ba4750608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3740188Z frame #53: _PyEval_EvalFrameDefault + 0x2ba5 (0x562ba4737c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3740783Z frame #54: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3741364Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x562ba47353bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3741952Z frame #56: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3742548Z frame #57: _PyEval_EvalFrameDefault + 0x1340 (0x562ba47363f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3743164Z frame #58: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3743763Z frame #59: _PyEval_EvalFrameDefault + 0x30c (0x562ba47353bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3744351Z frame #60: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3744941Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x562ba4737c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3745535Z frame #62: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3745872Z 
2025-07-17T11:07:25.3745993Z SIGABRT(6), PID: 1777269, Thread 1777460: 
2025-07-17T11:07:25.3746873Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3747786Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3748380Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3748963Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3749857Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3750707Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3751263Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3751590Z 
2025-07-17T11:07:25.3751704Z SIGABRT(6), PID: 1777269, Thread 1777463: 
2025-07-17T11:07:25.3752377Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3753191Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3753729Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3754266Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3755080Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3755985Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3756605Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3806548Z 
2025-07-17T11:07:25.3806792Z SIGABRT(6), PID: 1777268, Thread 1777268: 
2025-07-17T11:07:25.3807719Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3808760Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3809624Z frame #2: pthread_kill + 0x12c (0x7fc4439059fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3810298Z frame #3: raise + 0x16 (0x7fc4438b1476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3811160Z frame #4: <unknown function> + 0x4feaa4d (0x7fc2a058da4d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/_C/libtriton.so)
2025-07-17T11:07:25.3812117Z frame #5: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3812845Z frame #6: <unknown function> + 0x7741f (0x7fc2bfb6341f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.3813685Z frame #7: <unknown function> + 0x76c8e (0x7fc2bfb62c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.3814414Z frame #8: <unknown function> + 0x6ad69 (0x7fc2bfb56d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.3815123Z frame #9: <unknown function> + 0x3f54db (0x7fc3507bb4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3815737Z frame #10: <unknown function> + 0x3e11d8 (0x7fc3507a71d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3816426Z frame #11: <unknown function> + 0xb2f86 (0x7fc350478f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3817101Z frame #12: <unknown function> + 0x8e501 (0x7fc350454501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3817841Z frame #13: c10::hip::device_synchronize() + 0x5a (0x7fc37b25b7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.3818820Z frame #14: <unknown function> + 0xd307d9 (0x7fc3997ae7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.3819562Z frame #15: <unknown function> + 0x139902 (0x55e699982902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3820212Z frame #16: _PyEval_EvalFrameDefault + 0x49b5 (0x55e699979a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3820902Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3821537Z frame #18: _PyEval_EvalFrameDefault + 0x49b5 (0x55e699979a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3822194Z frame #19: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3822823Z frame #20: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3823427Z frame #21: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3824019Z frame #22: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3824602Z frame #23: <unknown function> + 0x1474b2 (0x55e6999904b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3825195Z frame #24: PyObject_Call + 0xbc (0x55e699990e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3825821Z frame #25: <unknown function> + 0x1ff69d (0x55e699a4869d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3826469Z frame #26: _PyObject_MakeTpCall + 0x2d3 (0x55e69997e023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3827084Z frame #27: _PyEval_EvalFrameDefault + 0x4c16 (0x55e699979cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3827738Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3828451Z frame #29: _PyEval_EvalFrameDefault + 0x30c (0x55e6999753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3829111Z frame #30: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3829708Z frame #31: _PyEval_EvalFrameDefault + 0x30c (0x55e6999753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3830344Z frame #32: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3830975Z frame #33: _PyEval_EvalFrameDefault + 0x49b5 (0x55e699979a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3831745Z frame #34: _PyObject_FastCallDictTstate + 0xd0 (0x55e69997d4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3832365Z frame #35: _PyObject_Call_Prepend + 0x69 (0x55e69998eab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3832926Z frame #36: <unknown function> + 0x205949 (0x55e699a4e949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3833512Z frame #37: _PyObject_MakeTpCall + 0x2d3 (0x55e69997e023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3834137Z frame #38: _PyEval_EvalFrameDefault + 0x4c16 (0x55e699979cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3834766Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3835408Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55e6999753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3836031Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3836683Z frame #42: _PyEval_EvalFrameDefault + 0x30c (0x55e6999753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3837309Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3837904Z frame #44: _PyEval_EvalFrameDefault + 0x30c (0x55e6999753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3838528Z frame #45: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3839305Z frame #46: _PyEval_EvalFrameDefault + 0x1340 (0x55e6999763f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3839923Z frame #47: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3840704Z frame #48: <unknown function> + 0x7ab3f7 (0x7fc3992293f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.3841454Z frame #49: <unknown function> + 0x13bc88 (0x55e699984c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3842053Z frame #50: PyObject_Call + 0x20f (0x55e699990f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3842666Z frame #51: _PyEval_EvalFrameDefault + 0x58dd (0x55e69997a98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3843288Z frame #52: <unknown function> + 0x147608 (0x55e699990608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3843887Z frame #53: _PyEval_EvalFrameDefault + 0x2ba5 (0x55e699977c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3844491Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3845097Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55e6999753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3845707Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3846382Z frame #57: _PyEval_EvalFrameDefault + 0x1340 (0x55e6999763f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3846978Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3847556Z frame #59: _PyEval_EvalFrameDefault + 0x30c (0x55e6999753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3848150Z frame #60: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3848822Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55e699977c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3849525Z frame #62: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3849867Z 
2025-07-17T11:07:25.3850014Z SIGABRT(6), PID: 1777269, Thread 1777466: 
2025-07-17T11:07:25.3850762Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3851585Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3852238Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3852785Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3853663Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3854517Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3855125Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3855497Z 
2025-07-17T11:07:25.3855644Z SIGABRT(6), PID: 1777268, Thread 1777375: 
2025-07-17T11:07:25.3856458Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3857307Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3857869Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3858417Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3859280Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3860251Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3860816Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3861160Z 
2025-07-17T11:07:25.3861270Z SIGABRT(6), PID: 1777269, Thread 1777469: 
2025-07-17T11:07:25.3861996Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3862926Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3863491Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3864075Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3864960Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3865777Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3866317Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3866737Z SIGABRT(6), PID: 1777268, Thread 1777377: 
2025-07-17T11:07:25.3867393Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3868273Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3868794Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3875004Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3875931Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3876785Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3877337Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3877654Z 
2025-07-17T11:07:25.3877744Z 
2025-07-17T11:07:25.3877874Z SIGABRT(6), PID: 1777269, Thread 1777472: 
2025-07-17T11:07:25.3878578Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3879403Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3879958Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3880511Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3881334Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3882260Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3882823Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3883142Z 
2025-07-17T11:07:25.3883275Z SIGABRT(6), PID: 1777268, Thread 1777380: 
2025-07-17T11:07:25.3883971Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3884820Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3885386Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3886031Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3886868Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3887724Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3888296Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3888630Z 
2025-07-17T11:07:25.3888772Z SIGABRT(6), PID: 1777269, Thread 1777475: 
2025-07-17T11:07:25.3889479Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3890290Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3890836Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3891439Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3892297Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3893146Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3893714Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3894091Z 
2025-07-17T11:07:25.3894200Z SIGABRT(6), PID: 1777268, Thread 1777382: 
2025-07-17T11:07:25.3894966Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3895796Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3896503Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3897105Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3897968Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3898898Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3899472Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3899936Z SIGABRT(6), PID: 1777269, Thread 1777478: 
2025-07-17T11:07:25.3900685Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3901522Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3902080Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3902625Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3903498Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3904348Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3904915Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3905241Z 
2025-07-17T11:07:25.3905245Z 
2025-07-17T11:07:25.3905461Z SIGABRT(6), PID: 1777269, Thread 1777481: 
2025-07-17T11:07:25.3906161Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3907070Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3907641Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3908240Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3909093Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3910023Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3910639Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3910964Z 
2025-07-17T11:07:25.3911088Z SIGABRT(6), PID: 1777268, Thread 1777384: 
2025-07-17T11:07:25.3911777Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3912615Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3913224Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3913780Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3914607Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3915471Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3916110Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3916454Z 
2025-07-17T11:07:25.3916576Z SIGABRT(6), PID: 1777269, Thread 1777484: 
2025-07-17T11:07:25.3917386Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3918202Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3918751Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3919381Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3920223Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3921092Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3921697Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3922053Z 
2025-07-17T11:07:25.3922172Z SIGABRT(6), PID: 1777268, Thread 1777386: 
2025-07-17T11:07:25.3922873Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3923681Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3924224Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3924795Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3925680Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3926523Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3927179Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3927533Z 
2025-07-17T11:07:25.3927655Z SIGABRT(6), PID: 1777269, Thread 1777487: 
2025-07-17T11:07:25.3928349Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3929158Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3929725Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3930269Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3931135Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3931992Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3932561Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3932891Z 
2025-07-17T11:07:25.3933027Z SIGABRT(6), PID: 1777268, Thread 1777388: 
2025-07-17T11:07:25.3933716Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3934530Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3935093Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3935636Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3936639Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3937580Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3938160Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3938479Z 
2025-07-17T11:07:25.3938614Z SIGABRT(6), PID: 1777269, Thread 1777490: 
2025-07-17T11:07:25.3939307Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3940212Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3940754Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3941299Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3942121Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3942990Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3943544Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3943882Z 
2025-07-17T11:07:25.3943993Z SIGABRT(6), PID: 1777270, Thread 1777270: 
2025-07-17T11:07:25.3944691Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3945515Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3946053Z frame #2: pthread_kill + 0x12c (0x7fe821ae69fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3946564Z frame #3: raise + 0x16 (0x7fe821a92476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3947241Z frame #4: <unknown function> + 0x4feaa4d (0x7fe67e66da4d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/_C/libtriton.so)
2025-07-17T11:07:25.3948065Z frame #5: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3948659Z frame #6: <unknown function> + 0x77435 (0x7fe69dd43435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.3949277Z frame #7: <unknown function> + 0x76c8e (0x7fe69dd42c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.3949888Z frame #8: <unknown function> + 0x6ad69 (0x7fe69dd36d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.3950488Z frame #9: <unknown function> + 0x3f54db (0x7fe72e99b4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3951065Z frame #10: <unknown function> + 0x3e11d8 (0x7fe72e9871d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3951652Z frame #11: <unknown function> + 0xb2f86 (0x7fe72e658f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3952208Z frame #12: <unknown function> + 0x8e501 (0x7fe72e634501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.3952954Z frame #13: c10::hip::device_synchronize() + 0x5a (0x7fe75943b7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.3953893Z frame #14: <unknown function> + 0xd307d9 (0x7fe77798e7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.3954688Z frame #15: <unknown function> + 0x139902 (0x55eb06340902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3955303Z frame #16: _PyEval_EvalFrameDefault + 0x49b5 (0x55eb06337a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3956021Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3956679Z frame #18: _PyEval_EvalFrameDefault + 0x49b5 (0x55eb06337a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3957368Z frame #19: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3958047Z frame #20: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3958662Z frame #21: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3959304Z frame #22: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3959903Z frame #23: <unknown function> + 0x1474b2 (0x55eb0634e4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3960543Z frame #24: PyObject_Call + 0xbc (0x55eb0634ee4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3961150Z frame #25: <unknown function> + 0x1ff69d (0x55eb0640669d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3961791Z frame #26: _PyObject_MakeTpCall + 0x2d3 (0x55eb0633c023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3962398Z frame #27: _PyEval_EvalFrameDefault + 0x4c16 (0x55eb06337cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3963028Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3963630Z frame #29: _PyEval_EvalFrameDefault + 0x30c (0x55eb063333bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3964240Z frame #30: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3964868Z frame #31: _PyEval_EvalFrameDefault + 0x30c (0x55eb063333bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3965476Z frame #32: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3966091Z frame #33: _PyEval_EvalFrameDefault + 0x49b5 (0x55eb06337a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3966776Z frame #34: _PyObject_FastCallDictTstate + 0xd0 (0x55eb0633b4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3967413Z frame #35: _PyObject_Call_Prepend + 0x69 (0x55eb0634cab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3968227Z frame #36: <unknown function> + 0x205949 (0x55eb0640c949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3968899Z frame #37: _PyObject_MakeTpCall + 0x2d3 (0x55eb0633c023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3969520Z frame #38: _PyEval_EvalFrameDefault + 0x4c16 (0x55eb06337cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3970154Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3970784Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55eb063333bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3971450Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3972064Z frame #42: _PyEval_EvalFrameDefault + 0x30c (0x55eb063333bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3972667Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3973286Z frame #44: _PyEval_EvalFrameDefault + 0x30c (0x55eb063333bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3973912Z frame #45: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3974517Z frame #46: _PyEval_EvalFrameDefault + 0x1340 (0x55eb063343f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3975135Z frame #47: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3975985Z frame #48: <unknown function> + 0x7ab3f7 (0x7fe7774093f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.3976800Z frame #49: <unknown function> + 0x13bc88 (0x55eb06342c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3977344Z frame #50: PyObject_Call + 0x20f (0x55eb0634ef9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3978031Z frame #51: _PyEval_EvalFrameDefault + 0x58dd (0x55eb0633898d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3979089Z frame #52: <unknown function> + 0x147608 (0x55eb0634e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3979914Z frame #53: _PyEval_EvalFrameDefault + 0x2ba5 (0x55eb06335c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3980639Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3981491Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55eb063333bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3982485Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3983420Z frame #57: _PyEval_EvalFrameDefault + 0x1340 (0x55eb063343f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3984215Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3984949Z frame #59: _PyEval_EvalFrameDefault + 0x30c (0x55eb063333bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3985776Z frame #60: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3986555Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55eb06335c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3987369Z frame #62: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.3987759Z 
2025-07-17T11:07:25.3988013Z SIGABRT(6), PID: 1777268, Thread 1777390: 
2025-07-17T11:07:25.3988910Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3989921Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3990712Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3991517Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3992710Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.3993831Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3994595Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3995030Z 
2025-07-17T11:07:25.3995346Z SIGABRT(6), PID: 1777269, Thread 1777493: 
2025-07-17T11:07:25.3996212Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.3997196Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3997934Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3998712Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.3999686Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4000748Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4001461Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4001843Z 
2025-07-17T11:07:25.4002256Z SIGABRT(6), PID: 1777270, Thread 1777498: 
2025-07-17T11:07:25.4003079Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4004005Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4004880Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4005707Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4006876Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4007985Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4008833Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4009274Z 
2025-07-17T11:07:25.4009428Z SIGABRT(6), PID: 1777268, Thread 1777392: 
2025-07-17T11:07:25.4010227Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4011508Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4012371Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4013188Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4014363Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4015442Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4016248Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4016722Z 
2025-07-17T11:07:25.4016905Z SIGABRT(6), PID: 1777269, Thread 1777496: 
2025-07-17T11:07:25.4017869Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4018882Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4019679Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4020372Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4021587Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4022635Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4023431Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4023887Z 
2025-07-17T11:07:25.4024121Z SIGABRT(6), PID: 1777270, Thread 1777502: 
2025-07-17T11:07:25.4024969Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4025921Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4026717Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4027498Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4028460Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4029470Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4030348Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4030780Z 
2025-07-17T11:07:25.4031029Z SIGABRT(6), PID: 1777268, Thread 1777395: 
2025-07-17T11:07:25.4032023Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4032980Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4033754Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4034640Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4035824Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4036917Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4037679Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4038123Z 
2025-07-17T11:07:25.4038293Z SIGABRT(6), PID: 1777269, Thread 1777500: 
2025-07-17T11:07:25.4039141Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4040331Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4041036Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4041733Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4042828Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4043921Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4044684Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4045087Z 
2025-07-17T11:07:25.4045433Z SIGABRT(6), PID: 1777270, Thread 1777506: 
2025-07-17T11:07:25.4046310Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4047341Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4048106Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4048772Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4050100Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4191659Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4192285Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4192651Z 
2025-07-17T11:07:25.4192806Z SIGABRT(6), PID: 1777268, Thread 1777398: 
2025-07-17T11:07:25.4193572Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4194442Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4195009Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4195573Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4196460Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4197469Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4198165Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4198491Z 
2025-07-17T11:07:25.4198673Z SIGABRT(6), PID: 1777269, Thread 1777504: 
2025-07-17T11:07:25.4199453Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4200314Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4200962Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4201495Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4202344Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4203194Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4203775Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4204088Z 
2025-07-17T11:07:25.4204221Z SIGABRT(6), PID: 1777270, Thread 1777510: 
2025-07-17T11:07:25.4204892Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4205790Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4206367Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4206878Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4207699Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4208640Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4209184Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4209507Z 
2025-07-17T11:07:25.4209613Z SIGABRT(6), PID: 1777269, Thread 1777509: 
2025-07-17T11:07:25.4210285Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4211104Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4211644Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4212171Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4213006Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4213821Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4214365Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4214692Z 
2025-07-17T11:07:25.4214791Z SIGABRT(6), PID: 1777268, Thread 1777400: 
2025-07-17T11:07:25.4215469Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4216479Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4217017Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4217623Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4218528Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4219461Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4220084Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4220404Z 
2025-07-17T11:07:25.4220623Z SIGABRT(6), PID: 1777270, Thread 1777514: 
2025-07-17T11:07:25.4221342Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4222247Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4222814Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4223387Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4224239Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4225103Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4225708Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4226039Z 
2025-07-17T11:07:25.4226196Z SIGABRT(6), PID: 1777269, Thread 1777513: 
2025-07-17T11:07:25.4226951Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4227820Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4228415Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4229078Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4229953Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4230766Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4231318Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4231710Z 
2025-07-17T11:07:25.4231807Z SIGABRT(6), PID: 1777268, Thread 1777403: 
2025-07-17T11:07:25.4232491Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4233381Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4233961Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4234535Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4235390Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4236204Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4236741Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4237068Z 
2025-07-17T11:07:25.4237177Z SIGABRT(6), PID: 1777270, Thread 1777518: 
2025-07-17T11:07:25.4237836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4238695Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4239298Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4239827Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4240644Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4241523Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4242074Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4242401Z 
2025-07-17T11:07:25.4242494Z SIGABRT(6), PID: 1777269, Thread 1777517: 
2025-07-17T11:07:25.4243174Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4243958Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4244507Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4245020Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4245832Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4246658Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4247213Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4247532Z 
2025-07-17T11:07:25.4247655Z SIGABRT(6), PID: 1777268, Thread 1777406: 
2025-07-17T11:07:25.4248332Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4249210Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4249746Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4250268Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4251105Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4251904Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4252442Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4252753Z 
2025-07-17T11:07:25.4252885Z SIGABRT(6), PID: 1777270, Thread 1777522: 
2025-07-17T11:07:25.4253544Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4254360Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4254863Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4255408Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4256319Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4257146Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4257677Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4258076Z 
2025-07-17T11:07:25.4258176Z SIGABRT(6), PID: 1777269, Thread 1777521: 
2025-07-17T11:07:25.4258947Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4259731Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4260267Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4260868Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4261678Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4262501Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4263037Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4263373Z 
2025-07-17T11:07:25.4263480Z SIGABRT(6), PID: 1777268, Thread 1777409: 
2025-07-17T11:07:25.4264154Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4264938Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4265474Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4266001Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4266795Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4267639Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4268161Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4268707Z SIGABRT(6), PID: 1777270, Thread 1777527: 
2025-07-17T11:07:25.4269371Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4270147Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4270675Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4271190Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4272005Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4272826Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4273370Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4273782Z SIGABRT(6), PID: 1777269, Thread 1777525: 
2025-07-17T11:07:25.4274443Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4275226Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4275748Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4276276Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4277168Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4277984Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4278572Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4278901Z 
2025-07-17T11:07:25.4278905Z 
2025-07-17T11:07:25.4278908Z 
2025-07-17T11:07:25.4279015Z SIGABRT(6), PID: 1777268, Thread 1777412: 
2025-07-17T11:07:25.4279685Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4280525Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4281051Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4281582Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4282398Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4283217Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4283751Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4284092Z 
2025-07-17T11:07:25.4284185Z SIGABRT(6), PID: 1777269, Thread 1777529: 
2025-07-17T11:07:25.4284852Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4285630Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4286167Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4286695Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4287517Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4288412Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4288931Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4289250Z 
2025-07-17T11:07:25.4289351Z SIGABRT(6), PID: 1777270, Thread 1777531: 
2025-07-17T11:07:25.4290036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4290800Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4291334Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4291851Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4292680Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4293487Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4294019Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4294323Z 
2025-07-17T11:07:25.4294436Z SIGABRT(6), PID: 1777268, Thread 1777415: 
2025-07-17T11:07:25.4295099Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4295892Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4296548Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4297165Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4297968Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4298784Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4299323Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4299735Z 
2025-07-17T11:07:25.4299833Z SIGABRT(6), PID: 1777269, Thread 1777533: 
2025-07-17T11:07:25.4300493Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4301264Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4301797Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4302321Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4303126Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4303936Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4304482Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4304807Z 
2025-07-17T11:07:25.4304902Z SIGABRT(6), PID: 1777270, Thread 1777535: 
2025-07-17T11:07:25.4305566Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4306354Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4306976Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4307506Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4308325Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4309144Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4309695Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4310004Z 
2025-07-17T11:07:25.4310127Z SIGABRT(6), PID: 1777268, Thread 1777418: 
2025-07-17T11:07:25.4310790Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4311569Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4312099Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4312628Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4313456Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4314294Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4314831Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4315147Z 
2025-07-17T11:07:25.4315253Z SIGABRT(6), PID: 1777268, Thread 1777421: 
2025-07-17T11:07:25.4316000Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4316865Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4317389Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4317921Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4318732Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4319633Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4320179Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4320489Z 
2025-07-17T11:07:25.4320599Z SIGABRT(6), PID: 1777269, Thread 1777537: 
2025-07-17T11:07:25.4321268Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4322048Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4322570Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4323109Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4323914Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4324732Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4325284Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4325726Z SIGABRT(6), PID: 1777270, Thread 1777539: 
2025-07-17T11:07:25.4326491Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4327283Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4327829Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4328376Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4329213Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4330037Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4330590Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4330906Z 
2025-07-17T11:07:25.4330911Z 
2025-07-17T11:07:25.4331043Z SIGABRT(6), PID: 1777268, Thread 1777424: 
2025-07-17T11:07:25.4331723Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4332491Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4333016Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4333554Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4334377Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4335292Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4335853Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4336234Z 
2025-07-17T11:07:25.4336433Z SIGABRT(6), PID: 1777269, Thread 1777541: 
2025-07-17T11:07:25.4337126Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4337906Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4338514Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4339026Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4339826Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4340657Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4341208Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4341545Z 
2025-07-17T11:07:25.4341651Z SIGABRT(6), PID: 1777270, Thread 1777542: 
2025-07-17T11:07:25.4342338Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4343132Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4343682Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4344223Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4345058Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4345882Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4346528Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4346864Z 
2025-07-17T11:07:25.4346968Z SIGABRT(6), PID: 1777268, Thread 1777427: 
2025-07-17T11:07:25.4347640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4348439Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4348986Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4349519Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4350349Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4351179Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4351728Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4352043Z 
2025-07-17T11:07:25.4352160Z SIGABRT(6), PID: 1777269, Thread 1777545: 
2025-07-17T11:07:25.4352850Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4353645Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4354187Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4354715Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4355643Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4356464Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4356983Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4357288Z 
2025-07-17T11:07:25.4357391Z SIGABRT(6), PID: 1777270, Thread 1777547: 
2025-07-17T11:07:25.4358142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4358934Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4359448Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4359994Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4360812Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4361605Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4362127Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4362443Z 
2025-07-17T11:07:25.4362540Z SIGABRT(6), PID: 1777268, Thread 1777429: 
2025-07-17T11:07:25.4363190Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4363955Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4364463Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4364986Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4365889Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4366718Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4367270Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4367614Z 
2025-07-17T11:07:25.4367725Z SIGABRT(6), PID: 1777269, Thread 1777549: 
2025-07-17T11:07:25.4368409Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4369208Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4369755Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4370294Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4371121Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4371930Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4372476Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4372798Z 
2025-07-17T11:07:25.4372925Z SIGABRT(6), PID: 1777270, Thread 1777551: 
2025-07-17T11:07:25.4373600Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4374464Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4375065Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4375600Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4376476Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4377372Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4377915Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4378230Z 
2025-07-17T11:07:25.4378345Z SIGABRT(6), PID: 1777268, Thread 1777432: 
2025-07-17T11:07:25.4379025Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4379821Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4380334Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4380864Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4381675Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4382508Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4383041Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4383378Z 
2025-07-17T11:07:25.4383474Z SIGABRT(6), PID: 1777269, Thread 1777553: 
2025-07-17T11:07:25.4384155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4385039Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4385559Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4386090Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4386914Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4387734Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4388278Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4388609Z 
2025-07-17T11:07:25.4388703Z SIGABRT(6), PID: 1777270, Thread 1777555: 
2025-07-17T11:07:25.4389372Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4390168Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4390713Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4391239Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4392067Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4392887Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4393440Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4393757Z 
2025-07-17T11:07:25.4393954Z SIGABRT(6), PID: 1777268, Thread 1777435: 
2025-07-17T11:07:25.4394715Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4395508Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4396041Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4396583Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4397469Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4398300Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4398824Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4399137Z 
2025-07-17T11:07:25.4399258Z SIGABRT(6), PID: 1777269, Thread 1777557: 
2025-07-17T11:07:25.4399930Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4400722Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4401256Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4401790Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4402606Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4403420Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4403955Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4404359Z 
2025-07-17T11:07:25.4404461Z SIGABRT(6), PID: 1777270, Thread 1777558: 
2025-07-17T11:07:25.4405129Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4405912Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4406453Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4406979Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4407803Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4408624Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4409172Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4409499Z 
2025-07-17T11:07:25.4409604Z SIGABRT(6), PID: 1777268, Thread 1777438: 
2025-07-17T11:07:25.4410286Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4411091Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4411619Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4412134Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4413013Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4413839Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4414460Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4414779Z 
2025-07-17T11:07:25.4414899Z SIGABRT(6), PID: 1777269, Thread 1777560: 
2025-07-17T11:07:25.4415565Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4416482Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4417021Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4417543Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4418363Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4419177Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4419710Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4420022Z 
2025-07-17T11:07:25.4420141Z SIGABRT(6), PID: 1777270, Thread 1777562: 
2025-07-17T11:07:25.4420814Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4421604Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4422128Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4422654Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4423475Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4424385Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4424918Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4425242Z 
2025-07-17T11:07:25.4425347Z SIGABRT(6), PID: 1777268, Thread 1777439: 
2025-07-17T11:07:25.4426024Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4426813Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4427335Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4427871Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4428695Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4429512Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4430050Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4430480Z SIGABRT(6), PID: 1777269, Thread 1777563: 
2025-07-17T11:07:25.4431167Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4431945Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4432483Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4433073Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4433946Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4434774Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4435315Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4435707Z 
2025-07-17T11:07:25.4435710Z 
2025-07-17T11:07:25.4435825Z SIGABRT(6), PID: 1777270, Thread 1777565: 
2025-07-17T11:07:25.4436503Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4437297Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4438099Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4438873Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4439759Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4440752Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4441408Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4441777Z 
2025-07-17T11:07:25.4441966Z SIGABRT(6), PID: 1777268, Thread 1777441: 
2025-07-17T11:07:25.4442769Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4443661Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4444346Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4445070Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4445961Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4446976Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4447720Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4448060Z 
2025-07-17T11:07:25.4448275Z SIGABRT(6), PID: 1777269, Thread 1777566: 
2025-07-17T11:07:25.4449091Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4450000Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4450713Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4451337Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4452249Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4453278Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4453916Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4454260Z 
2025-07-17T11:07:25.4454496Z SIGABRT(6), PID: 1777270, Thread 1777568: 
2025-07-17T11:07:25.4455492Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4456631Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4457353Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4457980Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4458915Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4460008Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4460659Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4461002Z 
2025-07-17T11:07:25.4461214Z SIGABRT(6), PID: 1777268, Thread 1777443: 
2025-07-17T11:07:25.4462022Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4462899Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4463693Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4464326Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4465211Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4466213Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4466804Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4467187Z 
2025-07-17T11:07:25.4467401Z SIGABRT(6), PID: 1777270, Thread 1777571: 
2025-07-17T11:07:25.4468162Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4469135Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4469807Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4470469Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4471377Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4472492Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4473129Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4473519Z 
2025-07-17T11:07:25.4473632Z SIGABRT(6), PID: 1777269, Thread 1777569: 
2025-07-17T11:07:25.4474489Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4475361Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4476044Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4476688Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4477592Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4478579Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4479292Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4479827Z 
2025-07-17T11:07:25.4480004Z SIGABRT(6), PID: 1777268, Thread 1777445: 
2025-07-17T11:07:25.4480857Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4481150Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4481489Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4481749Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4482369Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4482685Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4482929Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4482938Z 
2025-07-17T11:07:25.4483130Z SIGABRT(6), PID: 1777269, Thread 1777572: 
2025-07-17T11:07:25.4483634Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4483967Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4484231Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4484531Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4485065Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4485358Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4485670Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4485675Z 
2025-07-17T11:07:25.4485932Z SIGABRT(6), PID: 1777270, Thread 1777574: 
2025-07-17T11:07:25.4486461Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4486876Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4487122Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4487384Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4488039Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4488316Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4488611Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4488616Z 
2025-07-17T11:07:25.4488743Z SIGABRT(6), PID: 1777268, Thread 1777447: 
2025-07-17T11:07:25.4489268Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4489574Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4489898Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4490133Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4490785Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4491103Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4491465Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4491470Z 
2025-07-17T11:07:25.4491629Z SIGABRT(6), PID: 1777269, Thread 1777575: 
2025-07-17T11:07:25.4492177Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4492494Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4492775Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4493020Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4493680Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4494047Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4494336Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4494466Z SIGABRT(6), PID: 1777270, Thread 1777577: 
2025-07-17T11:07:25.4495018Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4495325Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4495629Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4495865Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4496650Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4496898Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4497237Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4497396Z SIGABRT(6), PID: 1777268, Thread 1777449: 
2025-07-17T11:07:25.4497944Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4498252Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4498498Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4498819Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4499377Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4499668Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4499933Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4499940Z 
2025-07-17T11:07:25.4499946Z 
2025-07-17T11:07:25.4499949Z 
2025-07-17T11:07:25.4500120Z SIGABRT(6), PID: 1777268, Thread 1777451: 
2025-07-17T11:07:25.4500604Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4501175Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4501430Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4501823Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4502380Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4502665Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4503070Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4503075Z 
2025-07-17T11:07:25.4503291Z SIGABRT(6), PID: 1777269, Thread 1777578: 
2025-07-17T11:07:25.4503800Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4504088Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4504333Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4504656Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4505234Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4505526Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4505774Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4505824Z 
2025-07-17T11:07:25.4505954Z SIGABRT(6), PID: 1777270, Thread 1777580: 
2025-07-17T11:07:25.4506551Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4506937Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4507224Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4507463Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4508032Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4508387Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4508781Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4508785Z 
2025-07-17T11:07:25.4508916Z SIGABRT(6), PID: 1777269, Thread 1777581: 
2025-07-17T11:07:25.4509468Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4509715Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4509983Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4510297Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4510898Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4511144Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4511434Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4511439Z 
2025-07-17T11:07:25.4511660Z SIGABRT(6), PID: 1777268, Thread 1777453: 
2025-07-17T11:07:25.4512320Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4512596Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4512882Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4513116Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4513776Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4514003Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4514372Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4514380Z 
2025-07-17T11:07:25.4514511Z SIGABRT(6), PID: 1777270, Thread 1777584: 
2025-07-17T11:07:25.4515077Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4515436Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4515702Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4516068Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4516607Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4516917Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4517172Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4517247Z 
2025-07-17T11:07:25.4517414Z SIGABRT(6), PID: 1777268, Thread 1777455: 
2025-07-17T11:07:25.4517993Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4518304Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4518567Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4518850Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4519379Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4519722Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4519988Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4519996Z 
2025-07-17T11:07:25.4520190Z SIGABRT(6), PID: 1777269, Thread 1777583: 
2025-07-17T11:07:25.4520697Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4520980Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4521204Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4521565Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4522116Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4522596Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4522911Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4522916Z 
2025-07-17T11:07:25.4523073Z SIGABRT(6), PID: 1777270, Thread 1777588: 
2025-07-17T11:07:25.4523653Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4524061Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4524309Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4524591Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4525122Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4525490Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4525760Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4525807Z 
2025-07-17T11:07:25.4525936Z SIGABRT(6), PID: 1777269, Thread 1777587: 
2025-07-17T11:07:25.4526482Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4526729Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4527079Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4527337Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4527922Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4528249Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4528562Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4528567Z 
2025-07-17T11:07:25.4528681Z SIGABRT(6), PID: 1777268, Thread 1777457: 
2025-07-17T11:07:25.4529461Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4529718Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4530002Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4530260Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4530829Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4531138Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4531451Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4531455Z 
2025-07-17T11:07:25.4531584Z SIGABRT(6), PID: 1777269, Thread 1777590: 
2025-07-17T11:07:25.4532151Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4532397Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4532724Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4533053Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4533717Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4533968Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4534255Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4534319Z 
2025-07-17T11:07:25.4534436Z SIGABRT(6), PID: 1777270, Thread 1777591: 
2025-07-17T11:07:25.4535072Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4535383Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4535631Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4535975Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4536502Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4537006Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4537283Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4537290Z 
2025-07-17T11:07:25.4537466Z SIGABRT(6), PID: 1777268, Thread 1777459: 
2025-07-17T11:07:25.4537978Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4538252Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4538554Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4538979Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4539521Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4539812Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4540063Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4540067Z 
2025-07-17T11:07:25.4540284Z SIGABRT(6), PID: 1777269, Thread 1777592: 
2025-07-17T11:07:25.4540836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4541127Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4541370Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4541652Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4542171Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4542559Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4542809Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4542983Z SIGABRT(6), PID: 1777268, Thread 1777462: 
2025-07-17T11:07:25.4543486Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4543981Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4544368Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4544676Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4545215Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4545598Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4545843Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4545875Z 
2025-07-17T11:07:25.4545878Z 
2025-07-17T11:07:25.4546074Z SIGABRT(6), PID: 1777270, Thread 1777594: 
2025-07-17T11:07:25.4546649Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4546903Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4547209Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4547445Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4548069Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4548340Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4548630Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4548634Z 
2025-07-17T11:07:25.4548781Z SIGABRT(6), PID: 1777268, Thread 1777465: 
2025-07-17T11:07:25.4549334Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4549636Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4550005Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4550243Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4550971Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4551217Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4551494Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4551498Z 
2025-07-17T11:07:25.4551687Z SIGABRT(6), PID: 1777270, Thread 1777595: 
2025-07-17T11:07:25.4552269Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4552537Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4552821Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4553057Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4553671Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4553934Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4554320Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4554326Z 
2025-07-17T11:07:25.4554504Z SIGABRT(6), PID: 1777269, Thread 1777597: 
2025-07-17T11:07:25.4555073Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4555414Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4555698Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4556072Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4556613Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4556900Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4557134Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4557141Z 
2025-07-17T11:07:25.4557421Z SIGABRT(6), PID: 1777268, Thread 1777468: 
2025-07-17T11:07:25.4558052Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4558344Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4558584Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4558844Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4559458Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4559770Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4560100Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4560105Z 
2025-07-17T11:07:25.4560278Z SIGABRT(6), PID: 1777269, Thread 1777600: 
2025-07-17T11:07:25.4560810Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4561141Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4561405Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4561683Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4562221Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4562528Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4703824Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4703852Z 
2025-07-17T11:07:25.4704057Z SIGABRT(6), PID: 1777270, Thread 1777598: 
2025-07-17T11:07:25.4704612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4704860Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4705080Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4705301Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4706016Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4706350Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4706579Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4706584Z 
2025-07-17T11:07:25.4706696Z SIGABRT(6), PID: 1777268, Thread 1777471: 
2025-07-17T11:07:25.4707197Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4707525Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4707737Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4707953Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4708476Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4708696Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4708912Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4708917Z 
2025-07-17T11:07:25.4709022Z SIGABRT(6), PID: 1777269, Thread 1777603: 
2025-07-17T11:07:25.4709501Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4709719Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4709922Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4710132Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4710728Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4710943Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4711157Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4711163Z 
2025-07-17T11:07:25.4711264Z SIGABRT(6), PID: 1777270, Thread 1777601: 
2025-07-17T11:07:25.4711739Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4711965Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4712177Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4712388Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4712891Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4713106Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4713319Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4713326Z 
2025-07-17T11:07:25.4713425Z SIGABRT(6), PID: 1777268, Thread 1777474: 
2025-07-17T11:07:25.4713894Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4714106Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4714377Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4714648Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4715162Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4715374Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4715650Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4715660Z 
2025-07-17T11:07:25.4715756Z SIGABRT(6), PID: 1777269, Thread 1777606: 
2025-07-17T11:07:25.4716239Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4716446Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4716663Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4716862Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4717356Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4717562Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4717786Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4717790Z 
2025-07-17T11:07:25.4717889Z SIGABRT(6), PID: 1777270, Thread 1777604: 
2025-07-17T11:07:25.4718387Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4718600Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4718886Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4719090Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4719607Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4719819Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4720034Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4720127Z SIGABRT(6), PID: 1777268, Thread 1777477: 
2025-07-17T11:07:25.4720603Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4720816Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4721021Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4721220Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4721720Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4721928Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4722141Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4722145Z 
2025-07-17T11:07:25.4722149Z 
2025-07-17T11:07:25.4722241Z SIGABRT(6), PID: 1777269, Thread 1777609: 
2025-07-17T11:07:25.4722776Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4723044Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4723255Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4723450Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4724036Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4724244Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4724457Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4724461Z 
2025-07-17T11:07:25.4724555Z SIGABRT(6), PID: 1777268, Thread 1777480: 
2025-07-17T11:07:25.4725030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4725237Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4725445Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4725638Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4726136Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4726341Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4726557Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4726649Z SIGABRT(6), PID: 1777270, Thread 1777607: 
2025-07-17T11:07:25.4727212Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4727418Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4727625Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4727823Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4728325Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4728540Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4728752Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4728758Z 
2025-07-17T11:07:25.4728761Z 
2025-07-17T11:07:25.4728860Z SIGABRT(6), PID: 1777269, Thread 1777612: 
2025-07-17T11:07:25.4729325Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4729534Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4729738Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4729943Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4730441Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4730718Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4730991Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4730996Z 
2025-07-17T11:07:25.4731094Z SIGABRT(6), PID: 1777268, Thread 1777483: 
2025-07-17T11:07:25.4731566Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4731776Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4732043Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4732243Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4732735Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4732945Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4733155Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4733159Z 
2025-07-17T11:07:25.4733254Z SIGABRT(6), PID: 1777270, Thread 1777610: 
2025-07-17T11:07:25.4733719Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4733931Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4734136Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4734337Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4734829Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4735111Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4735319Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4735324Z 
2025-07-17T11:07:25.4735421Z SIGABRT(6), PID: 1777269, Thread 1777615: 
2025-07-17T11:07:25.4735887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4736160Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4736363Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4736562Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4737053Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4737267Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4737475Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4737484Z 
2025-07-17T11:07:25.4737574Z SIGABRT(6), PID: 1777268, Thread 1777486: 
2025-07-17T11:07:25.4738044Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4738250Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4738805Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4739426Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4740323Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4741130Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4741654Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4741963Z 
2025-07-17T11:07:25.4742143Z SIGABRT(6), PID: 1777270, Thread 1777613: 
2025-07-17T11:07:25.4742809Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4743585Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4744098Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4744610Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4745416Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4746211Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4746733Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4747050Z 
2025-07-17T11:07:25.4747158Z SIGABRT(6), PID: 1777269, Thread 1777618: 
2025-07-17T11:07:25.4747815Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4748621Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4749133Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4749726Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4750522Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4751311Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4751831Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4752133Z 
2025-07-17T11:07:25.4752235Z SIGABRT(6), PID: 1777268, Thread 1777489: 
2025-07-17T11:07:25.4752887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4753670Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4754189Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4754704Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4755501Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4756297Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4756823Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4757136Z 
2025-07-17T11:07:25.4757233Z SIGABRT(6), PID: 1777270, Thread 1777616: 
2025-07-17T11:07:25.4757884Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4758724Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4759307Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4759815Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4760615Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4761471Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4761985Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4762390Z SIGABRT(6), PID: 1777269, Thread 1777621: 
2025-07-17T11:07:25.4763045Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4763820Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4764334Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4764838Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4765634Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4766429Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4766955Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4767263Z 
2025-07-17T11:07:25.4767266Z 
2025-07-17T11:07:25.4767367Z SIGABRT(6), PID: 1777268, Thread 1777492: 
2025-07-17T11:07:25.4768019Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4768855Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4769359Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4769859Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4770649Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4771451Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4771964Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4772359Z SIGABRT(6), PID: 1777270, Thread 1777619: 
2025-07-17T11:07:25.4773011Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4773780Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4774288Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4774794Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4775585Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4776441Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4776953Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4777261Z 
2025-07-17T11:07:25.4777350Z 
2025-07-17T11:07:25.4777449Z SIGABRT(6), PID: 1777269, Thread 1777624: 
2025-07-17T11:07:25.4778191Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4778957Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4779460Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4780030Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4780818Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4781603Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4782123Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4782433Z 
2025-07-17T11:07:25.4782528Z SIGABRT(6), PID: 1777270, Thread 1777622: 
2025-07-17T11:07:25.4783175Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4783935Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4784441Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4784941Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4785740Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4786534Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4787048Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4787449Z 
2025-07-17T11:07:25.4787546Z SIGABRT(6), PID: 1777268, Thread 1777495: 
2025-07-17T11:07:25.4788194Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4788951Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4789459Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4789958Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4790739Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4791531Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4792048Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4792352Z 
2025-07-17T11:07:25.4792443Z SIGABRT(6), PID: 1777269, Thread 1777628: 
2025-07-17T11:07:25.4793089Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4793850Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4794377Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4794887Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4795760Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4796633Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4797177Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4797496Z 
2025-07-17T11:07:25.4797595Z SIGABRT(6), PID: 1777270, Thread 1777625: 
2025-07-17T11:07:25.4798266Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4799110Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4799623Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4800134Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4800948Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4801751Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4802273Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4802582Z 
2025-07-17T11:07:25.4802677Z SIGABRT(6), PID: 1777268, Thread 1777499: 
2025-07-17T11:07:25.4803338Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4804107Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4804625Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4805135Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4805928Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4806807Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4807332Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4807637Z 
2025-07-17T11:07:25.4807740Z SIGABRT(6), PID: 1777269, Thread 1777631: 
2025-07-17T11:07:25.4808401Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4809172Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4809686Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4810198Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4811000Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4811801Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4812323Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4812631Z 
2025-07-17T11:07:25.4812733Z SIGABRT(6), PID: 1777270, Thread 1777629: 
2025-07-17T11:07:25.4813384Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4814151Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4814740Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4815257Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4816198Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4817016Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4817539Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4817936Z 
2025-07-17T11:07:25.4818034Z SIGABRT(6), PID: 1777267, Thread 1777267: 
2025-07-17T11:07:25.4818686Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4819451Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4819962Z frame #2: pthread_kill + 0x12c (0x7f88aac349fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4820424Z frame #3: raise + 0x16 (0x7f88aabe0476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4821069Z frame #4: <unknown function> + 0x4feaa4d (0x7f87077bea4d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/_C/libtriton.so)
2025-07-17T11:07:25.4821754Z frame #5: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4822310Z frame #6: <unknown function> + 0x7744b (0x7f8726e9444b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.4822882Z frame #7: <unknown function> + 0x76c8e (0x7f8726e93c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.4823436Z frame #8: <unknown function> + 0x6ad69 (0x7f8726e87d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.4823988Z frame #9: <unknown function> + 0x3f54db (0x7f87b7aec4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.4824536Z frame #10: <unknown function> + 0x3e11d8 (0x7f87b7ad81d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.4825173Z frame #11: <unknown function> + 0xb2f86 (0x7f87b77a9f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.4825693Z frame #12: <unknown function> + 0x8e501 (0x7f87b7785501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.4826401Z frame #13: c10::hip::device_synchronize() + 0x5a (0x7f87e258c7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.4827267Z frame #14: <unknown function> + 0xd307d9 (0x7f8800adf7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.4827973Z frame #15: <unknown function> + 0x139902 (0x55d3af81b902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4828556Z frame #16: _PyEval_EvalFrameDefault + 0x49b5 (0x55d3af812a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4829142Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4829710Z frame #18: _PyEval_EvalFrameDefault + 0x49b5 (0x55d3af812a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4830288Z frame #19: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4830853Z frame #20: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4831422Z frame #21: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4831989Z frame #22: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4832550Z frame #23: <unknown function> + 0x1474b2 (0x55d3af8294b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4833074Z frame #24: PyObject_Call + 0xbc (0x55d3af829e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4833694Z frame #25: <unknown function> + 0x1ff69d (0x55d3af8e169d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4834254Z frame #26: _PyObject_MakeTpCall + 0x2d3 (0x55d3af817023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4834889Z frame #27: _PyEval_EvalFrameDefault + 0x4c16 (0x55d3af812cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4835467Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4836038Z frame #29: _PyEval_EvalFrameDefault + 0x30c (0x55d3af80e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4836672Z frame #30: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4837243Z frame #31: _PyEval_EvalFrameDefault + 0x30c (0x55d3af80e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4837809Z frame #32: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4838376Z frame #33: _PyEval_EvalFrameDefault + 0x49b5 (0x55d3af812a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4838984Z frame #34: _PyObject_FastCallDictTstate + 0xd0 (0x55d3af8164b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4839562Z frame #35: _PyObject_Call_Prepend + 0x69 (0x55d3af827ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4840109Z frame #36: <unknown function> + 0x205949 (0x55d3af8e7949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4840660Z frame #37: _PyObject_MakeTpCall + 0x2d3 (0x55d3af817023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4841235Z frame #38: _PyEval_EvalFrameDefault + 0x4c16 (0x55d3af812cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4841798Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4842362Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55d3af80e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4842928Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4843506Z frame #42: _PyEval_EvalFrameDefault + 0x30c (0x55d3af80e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4844155Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4844721Z frame #44: _PyEval_EvalFrameDefault + 0x30c (0x55d3af80e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4845289Z frame #45: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4845863Z frame #46: _PyEval_EvalFrameDefault + 0x1340 (0x55d3af80f3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4846430Z frame #47: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4847153Z frame #48: <unknown function> + 0x7ab3f7 (0x7f880055a3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.4847857Z frame #49: <unknown function> + 0x13bc88 (0x55d3af81dc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4848395Z frame #50: PyObject_Call + 0x20f (0x55d3af829f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4848962Z frame #51: _PyEval_EvalFrameDefault + 0x58dd (0x55d3af81398d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4849529Z frame #52: <unknown function> + 0x147608 (0x55d3af829608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4850095Z frame #53: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d3af810c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4850666Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4851242Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55d3af80e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4851816Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4852378Z frame #57: _PyEval_EvalFrameDefault + 0x1340 (0x55d3af80f3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4853046Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4853686Z frame #59: _PyEval_EvalFrameDefault + 0x30c (0x55d3af80e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4854254Z frame #60: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4854807Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d3af810c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4855445Z frame #62: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.4855775Z 
2025-07-17T11:07:25.4855883Z SIGABRT(6), PID: 1777268, Thread 1777503: 
2025-07-17T11:07:25.4857010Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4857812Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4858345Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4858878Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4859697Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4860513Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4861042Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4861366Z 
2025-07-17T11:07:25.4861470Z SIGABRT(6), PID: 1777270, Thread 1777633: 
2025-07-17T11:07:25.4862142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4862931Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4863560Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4864074Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4864879Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4865687Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4866221Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4866534Z 
2025-07-17T11:07:25.4866642Z SIGABRT(6), PID: 1777269, Thread 1777635: 
2025-07-17T11:07:25.4867316Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4868094Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4868615Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4869131Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4869940Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4870753Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4871289Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4871605Z 
2025-07-17T11:07:25.4871715Z SIGABRT(6), PID: 1777268, Thread 1777507: 
2025-07-17T11:07:25.4872471Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4873331Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4873870Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4874394Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4875281Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4876090Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4876616Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4876944Z 
2025-07-17T11:07:25.4877045Z SIGABRT(6), PID: 1777267, Thread 1777393: 
2025-07-17T11:07:25.4877717Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4878504Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4879029Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4879548Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4880360Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4881177Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4881710Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4882032Z 
2025-07-17T11:07:25.4882132Z SIGABRT(6), PID: 1777270, Thread 1777637: 
2025-07-17T11:07:25.4882870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4883652Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4884181Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4884697Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4885506Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4886321Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4886871Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4887186Z 
2025-07-17T11:07:25.4887305Z SIGABRT(6), PID: 1777269, Thread 1777640: 
2025-07-17T11:07:25.4887975Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4888750Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4889270Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4889779Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4890584Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4891457Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4891989Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4892362Z 
2025-07-17T11:07:25.4892472Z SIGABRT(6), PID: 1777268, Thread 1777511: 
2025-07-17T11:07:25.4893134Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4893914Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4894504Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4895024Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4895817Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4896686Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4897214Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4897533Z 
2025-07-17T11:07:25.4897630Z SIGABRT(6), PID: 1777267, Thread 1777396: 
2025-07-17T11:07:25.4898292Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4899075Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4899596Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4900110Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4900918Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4901818Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4902358Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4902679Z 
2025-07-17T11:07:25.4902774Z SIGABRT(6), PID: 1777270, Thread 1777641: 
2025-07-17T11:07:25.4903437Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4904213Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4904728Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4905249Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4906054Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4906866Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4907405Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4907713Z 
2025-07-17T11:07:25.4907828Z SIGABRT(6), PID: 1777269, Thread 1777644: 
2025-07-17T11:07:25.4908485Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4909263Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4909788Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4910385Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4911261Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4912088Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4912615Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4912926Z 
2025-07-17T11:07:25.4913034Z SIGABRT(6), PID: 1777268, Thread 1777515: 
2025-07-17T11:07:25.4913775Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4914552Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4915079Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4915593Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4916396Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4917208Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4917738Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4918059Z 
2025-07-17T11:07:25.4918155Z SIGABRT(6), PID: 1777267, Thread 1777401: 
2025-07-17T11:07:25.4918816Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4919595Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4920120Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4920751Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4921560Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4922365Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4922899Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4923221Z 
2025-07-17T11:07:25.4923315Z SIGABRT(6), PID: 1777270, Thread 1777645: 
2025-07-17T11:07:25.4923982Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4924760Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4925291Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4925811Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4926624Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4927435Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4927974Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4928281Z 
2025-07-17T11:07:25.4928392Z SIGABRT(6), PID: 1777268, Thread 1777519: 
2025-07-17T11:07:25.4929062Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4929913Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4930509Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4931031Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4931830Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4932705Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4933238Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4933546Z 
2025-07-17T11:07:25.4933661Z SIGABRT(6), PID: 1777269, Thread 1777648: 
2025-07-17T11:07:25.4934324Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4935106Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4935637Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4936202Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4936993Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4937811Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4938346Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4938671Z 
2025-07-17T11:07:25.4938773Z SIGABRT(6), PID: 1777267, Thread 1777404: 
2025-07-17T11:07:25.4939434Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4940298Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4940833Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4941352Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4942162Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4942972Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4943511Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4943834Z 
2025-07-17T11:07:25.4943932Z SIGABRT(6), PID: 1777270, Thread 1777649: 
2025-07-17T11:07:25.4944587Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4945366Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4945894Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4946409Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4947227Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4948044Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4948579Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4948888Z 
2025-07-17T11:07:25.4949084Z SIGABRT(6), PID: 1777268, Thread 1777523: 
2025-07-17T11:07:25.4949826Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4950605Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4951135Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4951868Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4952683Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4953494Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4954028Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4954345Z 
2025-07-17T11:07:25.4954457Z SIGABRT(6), PID: 1777269, Thread 1777652: 
2025-07-17T11:07:25.4955122Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4955896Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4956422Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4956925Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4957739Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4958548Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4959085Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4959478Z 
2025-07-17T11:07:25.4959584Z SIGABRT(6), PID: 1777267, Thread 1777407: 
2025-07-17T11:07:25.4960263Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4961053Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4961588Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4962109Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4962925Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4963739Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4964283Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4964605Z 
2025-07-17T11:07:25.4964701Z SIGABRT(6), PID: 1777270, Thread 1777653: 
2025-07-17T11:07:25.4965364Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4966143Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4966673Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4967191Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4968080Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4968912Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4969524Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4969839Z 
2025-07-17T11:07:25.4969946Z SIGABRT(6), PID: 1777268, Thread 1777526: 
2025-07-17T11:07:25.4970612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4971463Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4971991Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4972509Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4973314Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4974130Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4974663Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4974975Z 
2025-07-17T11:07:25.4975090Z SIGABRT(6), PID: 1777269, Thread 1777656: 
2025-07-17T11:07:25.4975758Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4976611Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4977137Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4977641Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4978461Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4979366Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4979896Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4980220Z 
2025-07-17T11:07:25.4980319Z SIGABRT(6), PID: 1777270, Thread 1777657: 
2025-07-17T11:07:25.4980987Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4981768Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4982290Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4982809Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4983618Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4984425Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4984957Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4985282Z 
2025-07-17T11:07:25.4985379Z SIGABRT(6), PID: 1777267, Thread 1777410: 
2025-07-17T11:07:25.4986037Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4986815Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4987431Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4987967Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4988850Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4989674Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4990210Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4990643Z 
2025-07-17T11:07:25.4990756Z SIGABRT(6), PID: 1777268, Thread 1777530: 
2025-07-17T11:07:25.4991435Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4992229Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4992762Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4993290Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4994114Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.4994936Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4995477Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4995793Z 
2025-07-17T11:07:25.4995908Z SIGABRT(6), PID: 1777269, Thread 1777660: 
2025-07-17T11:07:25.4996586Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.4997394Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4998019Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4998538Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.4999363Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5000188Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5000728Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5001156Z SIGABRT(6), PID: 1777270, Thread 1777661: 
2025-07-17T11:07:25.5001832Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5002616Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5003146Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5003669Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5004486Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5005306Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5005850Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5006268Z SIGABRT(6), PID: 1777267, Thread 1777413: 
2025-07-17T11:07:25.5007009Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5007794Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5008385Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5008911Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5009723Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5010605Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5011138Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5011553Z SIGABRT(6), PID: 1777268, Thread 1777534: 
2025-07-17T11:07:25.5012230Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5013017Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5013541Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5014045Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5014852Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5015668Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5016264Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5016574Z 
2025-07-17T11:07:25.5016578Z 
2025-07-17T11:07:25.5016582Z 
2025-07-17T11:07:25.5016585Z 
2025-07-17T11:07:25.5016697Z SIGABRT(6), PID: 1777269, Thread 1777664: 
2025-07-17T11:07:25.5017452Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5018234Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5018750Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5019257Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5020070Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5020880Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5021412Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5021739Z 
2025-07-17T11:07:25.5021840Z SIGABRT(6), PID: 1777270, Thread 1777665: 
2025-07-17T11:07:25.5022505Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5023282Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5023806Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5024328Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5025130Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5025940Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5026548Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5027044Z SIGABRT(6), PID: 1777268, Thread 1777538: 
2025-07-17T11:07:25.5027724Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5028500Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5029023Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5029615Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5030427Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5031230Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5031761Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5032078Z 
2025-07-17T11:07:25.5032082Z 
2025-07-17T11:07:25.5032186Z SIGABRT(6), PID: 1777267, Thread 1777416: 
2025-07-17T11:07:25.5032844Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5033621Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5034144Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5034657Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5035470Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5036279Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5036886Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5037206Z 
2025-07-17T11:07:25.5037313Z SIGABRT(6), PID: 1777268, Thread 1777543: 
2025-07-17T11:07:25.5037976Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5038764Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5039274Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5039787Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5040597Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5041408Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5041937Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5042256Z 
2025-07-17T11:07:25.5042351Z SIGABRT(6), PID: 1777270, Thread 1777668: 
2025-07-17T11:07:25.5043014Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5043793Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5044318Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5044839Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5045773Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5046603Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5047134Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5047448Z 
2025-07-17T11:07:25.5047542Z SIGABRT(6), PID: 1777269, Thread 1777671: 
2025-07-17T11:07:25.5048268Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5049050Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5049574Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5050094Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5050908Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5051724Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5052253Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5052564Z 
2025-07-17T11:07:25.5052673Z SIGABRT(6), PID: 1777267, Thread 1777419: 
2025-07-17T11:07:25.5053334Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5054118Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5054645Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5055160Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5056103Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5056908Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5057446Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5057452Z 
2025-07-17T11:07:25.5057553Z SIGABRT(6), PID: 1777269, Thread 1777678: 
2025-07-17T11:07:25.5058048Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5058262Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5058488Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5058697Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5059214Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5059426Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5059653Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5059658Z 
2025-07-17T11:07:25.5059758Z SIGABRT(6), PID: 1777268, Thread 1777546: 
2025-07-17T11:07:25.5060247Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5060542Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5060769Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5061065Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5061577Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5061787Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5062088Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5062092Z 
2025-07-17T11:07:25.5062195Z SIGABRT(6), PID: 1777270, Thread 1777672: 
2025-07-17T11:07:25.5062678Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5062893Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5063119Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5063320Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5063820Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5064032Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5064254Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5064258Z 
2025-07-17T11:07:25.5064355Z SIGABRT(6), PID: 1777267, Thread 1777422: 
2025-07-17T11:07:25.5064846Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5065146Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5065360Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5065579Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5066076Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5066294Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5066511Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5066514Z 
2025-07-17T11:07:25.5066621Z SIGABRT(6), PID: 1777269, Thread 1777682: 
2025-07-17T11:07:25.5067093Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5067308Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5067522Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5067736Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5068237Z frame #4: <unknown function> + 0x3475ab (0x7fb85ff305ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5068458Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5068674Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5068678Z 
2025-07-17T11:07:25.5068862Z SIGABRT(6), PID: 1777268, Thread 1777550: 
2025-07-17T11:07:25.5069396Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5069625Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5069836Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5070053Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5070614Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5070838Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5071048Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5071055Z 
2025-07-17T11:07:25.5071162Z SIGABRT(6), PID: 1777267, Thread 1777425: 
2025-07-17T11:07:25.5071634Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5071856Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5072067Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5072283Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5072781Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5073001Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5073218Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5073287Z 
2025-07-17T11:07:25.5073396Z SIGABRT(6), PID: 1777270, Thread 1777675: 
2025-07-17T11:07:25.5073871Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5074092Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5074306Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5074519Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5075024Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5075249Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5075464Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5075482Z 
2025-07-17T11:07:25.5075578Z SIGABRT(6), PID: 1777269, Thread 1777791: 
2025-07-17T11:07:25.5076066Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5076520Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fb77178f08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5076745Z frame #2: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5076946Z frame #3: pthread_kill + 0x12c (0x7fb86309a9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5077128Z frame #4: raise + 0x16 (0x7fb863046476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5077366Z frame #5: abort + 0xd3 (0x7fb86302c7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5077688Z frame #6: <unknown function> + 0x972cc (0x7fb6df3182cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5077930Z frame #7: <unknown function> + 0x95f47 (0x7fb6df316f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5078174Z frame #8: <unknown function> + 0x34b71 (0x7fb6df2b5b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5078391Z frame #9: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5078706Z frame #10: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5078711Z 
2025-07-17T11:07:25.5078810Z SIGABRT(6), PID: 1777268, Thread 1777554: 
2025-07-17T11:07:25.5079303Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5079522Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5079745Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5079951Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5080464Z frame #4: <unknown function> + 0x3475ab (0x7fc44079b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5080678Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5080905Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5081003Z SIGABRT(6), PID: 1777267, Thread 1777428: 
2025-07-17T11:07:25.5081484Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5081699Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5081984Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5082190Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5082706Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5082923Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5083149Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5083153Z 
2025-07-17T11:07:25.5083157Z 
2025-07-17T11:07:25.5083252Z SIGABRT(6), PID: 1777270, Thread 1777679: 
2025-07-17T11:07:25.5083736Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5083953Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5084174Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5084373Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5084878Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5085092Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5085317Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5085321Z 
2025-07-17T11:07:25.5085416Z SIGABRT(6), PID: 1777269, Thread 1780708: 
2025-07-17T11:07:25.5086012Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5086230Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5086416Z frame #2: __poll + 0x4f (0x7fb86311cbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5086831Z frame #3: <unknown function> + 0x489f (0x7fb862c2889f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.5087138Z frame #4: <unknown function> + 0x1483bc (0x562ba47513bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5087392Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5087639Z frame #6: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5087884Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5088134Z frame #8: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5088380Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x562ba4739a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5088628Z frame #10: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5088883Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x562ba4737c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5089136Z frame #12: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5089374Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5089606Z frame #14: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5089845Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5090146Z frame #16: <unknown function> + 0x147734 (0x562ba4750734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5090370Z frame #17: <unknown function> + 0x21ca82 (0x562ba4825a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5090604Z frame #18: <unknown function> + 0x21c9e4 (0x562ba48259e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5090821Z frame #19: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5091049Z frame #20: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5091147Z SIGABRT(6), PID: 1777268, Thread 1777792: 
2025-07-17T11:07:25.5091637Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5092098Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fc351ffa08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5092327Z frame #2: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5092524Z frame #3: pthread_kill + 0x12c (0x7fc4439059fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5092703Z frame #4: raise + 0x16 (0x7fc4438b1476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5092866Z frame #5: abort + 0xd3 (0x7fc4438977f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5093119Z frame #6: <unknown function> + 0x972cc (0x7fc2bfb832cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5093354Z frame #7: <unknown function> + 0x95f47 (0x7fc2bfb81f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5093597Z frame #8: <unknown function> + 0x34b71 (0x7fc2bfb20b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5093811Z frame #9: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5094128Z frame #10: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5094135Z 
2025-07-17T11:07:25.5094192Z 
2025-07-17T11:07:25.5094299Z SIGABRT(6), PID: 1777267, Thread 1777431: 
2025-07-17T11:07:25.5094808Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5095044Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5095325Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5095549Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5096116Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5096347Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5096567Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5096572Z 
2025-07-17T11:07:25.5096677Z SIGABRT(6), PID: 1777270, Thread 1777683: 
2025-07-17T11:07:25.5097154Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5097376Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5097591Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5097810Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5098315Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5098629Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5098845Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5098849Z 
2025-07-17T11:07:25.5098956Z SIGABRT(6), PID: 1777267, Thread 1777434: 
2025-07-17T11:07:25.5099429Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5099657Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5099869Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5100081Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5100586Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5100812Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5101029Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5101033Z 
2025-07-17T11:07:25.5101145Z SIGABRT(6), PID: 1777268, Thread 1781321: 
2025-07-17T11:07:25.5101617Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5101840Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5102015Z frame #2: __poll + 0x4f (0x7fc443987bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5102514Z frame #3: <unknown function> + 0x489f (0x7fc44349389f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.5102826Z frame #4: <unknown function> + 0x1483bc (0x55e6999913bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5103090Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5103323Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5103576Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5103882Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5104143Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55e699979a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5104384Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5104651Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55e699977c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5104886Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5105140Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5105369Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5105624Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5105849Z frame #16: <unknown function> + 0x147734 (0x55e699990734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5106087Z frame #17: <unknown function> + 0x21ca82 (0x55e699a65a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5106315Z frame #18: <unknown function> + 0x21c9e4 (0x55e699a659e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5106545Z frame #19: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5106768Z frame #20: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5106839Z 
2025-07-17T11:07:25.5106953Z SIGABRT(6), PID: 1777270, Thread 1777686: 
2025-07-17T11:07:25.5107435Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5107672Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5107889Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5108111Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5108612Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5108841Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5109061Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5109065Z 
2025-07-17T11:07:25.5109176Z SIGABRT(6), PID: 1777267, Thread 1777437: 
2025-07-17T11:07:25.5109662Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5109878Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5110102Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5110308Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5110891Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5111170Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5111407Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5111411Z 
2025-07-17T11:07:25.5111509Z SIGABRT(6), PID: 1777270, Thread 1777689: 
2025-07-17T11:07:25.5111996Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5112266Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5112487Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5112693Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5122856Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5123153Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5123393Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5123399Z 
2025-07-17T11:07:25.5123506Z SIGABRT(6), PID: 1777269, Thread 1781252: 
2025-07-17T11:07:25.5124015Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5124240Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5124454Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5124687Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fb8630982dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5125150Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fb79d4d7048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5125702Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fb79d4d9bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5125948Z frame #6: <unknown function> + 0xdc253 (0x7fb76f9fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5126162Z frame #7: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5126379Z frame #8: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5126383Z 
2025-07-17T11:07:25.5126479Z SIGABRT(6), PID: 1777267, Thread 1777440: 
2025-07-17T11:07:25.5126965Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5127175Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5127391Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5127594Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5128110Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5128320Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5128534Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5128628Z SIGABRT(6), PID: 1777270, Thread 1777694: 
2025-07-17T11:07:25.5129180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5129462Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5129675Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5129877Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5130379Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5130674Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5130887Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5130891Z 
2025-07-17T11:07:25.5130895Z 
2025-07-17T11:07:25.5130986Z SIGABRT(6), PID: 1777268, Thread 1781322: 
2025-07-17T11:07:25.5131458Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5131669Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5131872Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5132095Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fc4439032dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5132541Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fc37dd42048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5132977Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fc37dd44bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5133222Z frame #6: <unknown function> + 0xdc253 (0x7fc350266253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5133446Z frame #7: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5133735Z frame #8: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5133740Z 
2025-07-17T11:07:25.5133840Z SIGABRT(6), PID: 1777270, Thread 1777697: 
2025-07-17T11:07:25.5134331Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5134551Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5134755Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5134962Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5135473Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5135701Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5135991Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5135996Z 
2025-07-17T11:07:25.5136103Z SIGABRT(6), PID: 1777269, Thread 1781253: 
2025-07-17T11:07:25.5136581Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5136800Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5137006Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5137236Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fb8630982dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5137862Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fb79d4d1741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5138119Z frame #5: <unknown function> + 0xdc253 (0x7fb76f9fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5138330Z frame #6: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5138546Z frame #7: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5138717Z SIGABRT(6), PID: 1777267, Thread 1777442: 
2025-07-17T11:07:25.5139195Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5139401Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5139614Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5139819Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5140329Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5140538Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5140755Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5140761Z 
2025-07-17T11:07:25.5140764Z 
2025-07-17T11:07:25.5140854Z SIGABRT(6), PID: 1777269, Thread 1781255: 
2025-07-17T11:07:25.5141328Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5141533Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5141711Z frame #2: ioctl + 0x3f (0x7fb86311e94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5142040Z frame #3: <unknown function> + 0x13eef0 (0x7fb6df3bfef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5142279Z frame #4: <unknown function> + 0x1378c4 (0x7fb6df3b88c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5142512Z frame #5: <unknown function> + 0x95b2b (0x7fb6df316b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5142746Z frame #6: <unknown function> + 0x34b71 (0x7fb6df2b5b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5142956Z frame #7: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5143170Z frame #8: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5143174Z 
2025-07-17T11:07:25.5143264Z SIGABRT(6), PID: 1777270, Thread 1777699: 
2025-07-17T11:07:25.5143738Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5143948Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5144160Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5144364Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5144870Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5145083Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5145293Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5145297Z 
2025-07-17T11:07:25.5145463Z SIGABRT(6), PID: 1777267, Thread 1777444: 
2025-07-17T11:07:25.5145996Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5146216Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5146424Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5146630Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5147185Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5147397Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5147618Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5147625Z 
2025-07-17T11:07:25.5147720Z SIGABRT(6), PID: 1777268, Thread 1781323: 
2025-07-17T11:07:25.5148193Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5148401Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5148603Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5148833Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fc4439032dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5149308Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fc37dd3c741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5149545Z frame #5: <unknown function> + 0xdc253 (0x7fc350266253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5149753Z frame #6: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5150034Z frame #7: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5150039Z 
2025-07-17T11:07:25.5150129Z SIGABRT(6), PID: 1777270, Thread 1777703: 
2025-07-17T11:07:25.5150602Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5150809Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5151018Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5151220Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5151730Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5151937Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5152154Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5152158Z 
2025-07-17T11:07:25.5152251Z SIGABRT(6), PID: 1777268, Thread 1781324: 
2025-07-17T11:07:25.5152727Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5152935Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5153106Z frame #2: ioctl + 0x3f (0x7fc44398994f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5153343Z frame #3: <unknown function> + 0x13eef0 (0x7fc2bfc2aef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5153579Z frame #4: <unknown function> + 0x1378c4 (0x7fc2bfc238c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5154018Z frame #5: <unknown function> + 0x95b2b (0x7fc2bfb81b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5154318Z frame #6: <unknown function> + 0x34b71 (0x7fc2bfb20b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5154530Z frame #7: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5154746Z frame #8: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5154838Z SIGABRT(6), PID: 1777269, Thread 1781540: 
2025-07-17T11:07:25.5155373Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5155580Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5155750Z frame #2: __poll + 0x4f (0x7fb86311cbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5155965Z frame #3: <unknown function> + 0xd4540 (0x7fb72752f540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5156176Z frame #4: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5156387Z frame #5: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5156391Z 
2025-07-17T11:07:25.5156395Z 
2025-07-17T11:07:25.5156488Z SIGABRT(6), PID: 1777267, Thread 1777446: 
2025-07-17T11:07:25.5156955Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5157169Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5157380Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5157587Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5158089Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5158363Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5158579Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5158583Z 
2025-07-17T11:07:25.5158676Z SIGABRT(6), PID: 1777269, Thread 1781548: 
2025-07-17T11:07:25.5159145Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5159349Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5159519Z frame #2: __poll + 0x4f (0x7fb86311cbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5159733Z frame #3: <unknown function> + 0xd65a2 (0x7fb7275315a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5159942Z frame #4: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5160148Z frame #5: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5160241Z SIGABRT(6), PID: 1777270, Thread 1777708: 
2025-07-17T11:07:25.5160705Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5160914Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5161118Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5161323Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5161882Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5162179Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5162392Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5162488Z SIGABRT(6), PID: 1777268, Thread 1781542: 
2025-07-17T11:07:25.5162953Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5163233Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5163396Z frame #2: __poll + 0x4f (0x7fc443987bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5163614Z frame #3: <unknown function> + 0xd4540 (0x7fc307d9a540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5163821Z frame #4: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5164034Z frame #5: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5164127Z SIGABRT(6), PID: 1777267, Thread 1777448: 
2025-07-17T11:07:25.5164593Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5164800Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5165011Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5165210Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5165712Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5165923Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5166212Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5166216Z 
2025-07-17T11:07:25.5166220Z 
2025-07-17T11:07:25.5166223Z 
2025-07-17T11:07:25.5166226Z 
2025-07-17T11:07:25.5166319Z SIGABRT(6), PID: 1777269, Thread 1781559: 
2025-07-17T11:07:25.5166792Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5167002Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5167212Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5167412Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5167640Z frame #4: <unknown function> + 0xd1e7f (0x7fb72752ce7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5167851Z frame #5: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5168066Z frame #6: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5168070Z 
2025-07-17T11:07:25.5168167Z SIGABRT(6), PID: 1777268, Thread 1781550: 
2025-07-17T11:07:25.5168638Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5168845Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5169013Z frame #2: __poll + 0x4f (0x7fc443987bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5169226Z frame #3: <unknown function> + 0xd65a2 (0x7fc307d9c5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5169504Z frame #4: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5169718Z frame #5: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5169813Z 
2025-07-17T11:07:25.5169907Z SIGABRT(6), PID: 1777267, Thread 1777450: 
2025-07-17T11:07:25.5170377Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5170584Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5170859Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5171058Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5171567Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5171777Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5171993Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5171998Z 
2025-07-17T11:07:25.5172085Z SIGABRT(6), PID: 1777270, Thread 1777774: 
2025-07-17T11:07:25.5172558Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5172765Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5172973Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5173171Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5173676Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5173953Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5174167Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5174171Z 
2025-07-17T11:07:25.5174261Z SIGABRT(6), PID: 1777268, Thread 1781560: 
2025-07-17T11:07:25.5174726Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5174932Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5175141Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5175340Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5175560Z frame #4: <unknown function> + 0xd1e7f (0x7fc307d97e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5175769Z frame #5: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5176037Z frame #6: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5176041Z 
2025-07-17T11:07:25.5176132Z SIGABRT(6), PID: 1777269, Thread 1781566: 
2025-07-17T11:07:25.5176601Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5176807Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5177014Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5177218Z frame #3: <unknown function> + 0x9ca53 (0x7fb8630a0a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5177565Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x562ba471dbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5177871Z frame #5: <unknown function> + 0x1d314f (0x562ba47dc14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5178103Z frame #6: <unknown function> + 0x1d2f6b (0x562ba47dbf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5178325Z frame #7: <unknown function> + 0x6df36 (0x562ba4676f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5178574Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5178871Z frame #9: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5179120Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5179348Z frame #11: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5179588Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5179815Z frame #13: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5180058Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5180284Z frame #15: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5180525Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5180750Z frame #17: <unknown function> + 0x147734 (0x562ba4750734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5180975Z frame #18: <unknown function> + 0x21ca82 (0x562ba4825a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5181193Z frame #19: <unknown function> + 0x21c9e4 (0x562ba48259e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5181411Z frame #20: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5181627Z frame #21: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5181812Z SIGABRT(6), PID: 1777267, Thread 1777452: 
2025-07-17T11:07:25.5182283Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5182499Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5182707Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5182915Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5183422Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5183638Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5183852Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5183856Z 
2025-07-17T11:07:25.5183860Z 
2025-07-17T11:07:25.5183957Z SIGABRT(6), PID: 1777270, Thread 1777775: 
2025-07-17T11:07:25.5184422Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5184634Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5184837Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5185043Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5185597Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5185874Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5186088Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5186099Z 
2025-07-17T11:07:25.5186189Z SIGABRT(6), PID: 1777268, Thread 1781570: 
2025-07-17T11:07:25.5186655Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5186924Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5187130Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5187335Z frame #3: <unknown function> + 0x9ca53 (0x7fc44390ba53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5187593Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55e69995dbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5187832Z frame #5: <unknown function> + 0x1d314f (0x55e699a1c14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5188060Z frame #6: <unknown function> + 0x1d2f6b (0x55e699a1bf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5188279Z frame #7: <unknown function> + 0x6df36 (0x55e6998b6f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5188520Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5188745Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5188986Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5189212Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5189453Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5189678Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5189981Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5190206Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5190443Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5190663Z frame #17: <unknown function> + 0x147734 (0x55e699990734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5190888Z frame #18: <unknown function> + 0x21ca82 (0x55e699a65a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5191108Z frame #19: <unknown function> + 0x21c9e4 (0x55e699a659e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5191327Z frame #20: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5191539Z frame #21: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5191547Z 
2025-07-17T11:07:25.5191643Z SIGABRT(6), PID: 1777267, Thread 1777454: 
2025-07-17T11:07:25.5192108Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5192324Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5192531Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5192736Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5193235Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5193536Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5193812Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5193819Z 
2025-07-17T11:07:25.5193917Z SIGABRT(6), PID: 1777270, Thread 1777776: 
2025-07-17T11:07:25.5194382Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5194654Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5194856Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5195058Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5195553Z frame #4: <unknown function> + 0x3475ab (0x7fe81e97c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5195768Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5195974Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5195979Z 
2025-07-17T11:07:25.5196072Z SIGABRT(6), PID: 1777269, Thread 1781578: 
2025-07-17T11:07:25.5196534Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5196745Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5196908Z frame #2: read + 0x4c (0x7fb86311881c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5197096Z frame #3: _Py_read + 0x4c (0x562ba47e3a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5197324Z frame #4: <unknown function> + 0x208c3b (0x562ba4811c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5197562Z frame #5: <unknown function> + 0x14648d (0x562ba474f48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5197877Z frame #6: PyObject_VectorcallMethod + 0x85 (0x562ba475e215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5198103Z frame #7: <unknown function> + 0x208ae7 (0x562ba4811ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5198319Z frame #8: <unknown function> + 0x208472 (0x562ba4811472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5198546Z frame #9: <unknown function> + 0x1da6ee (0x562ba47e36ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5198769Z frame #10: <unknown function> + 0x1483bc (0x562ba47513bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5199016Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5199241Z frame #12: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5199484Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x562ba47353bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5199708Z frame #14: <unknown function> + 0x147734 (0x562ba4750734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5199961Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x562ba4737c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5200185Z frame #16: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5200425Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5200644Z frame #18: _PyFunction_Vectorcall + 0x6c (0x562ba47450ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5200880Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x562ba47357b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5201098Z frame #20: <unknown function> + 0x147734 (0x562ba4750734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5201390Z frame #21: <unknown function> + 0x21ca82 (0x562ba4825a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5201675Z frame #22: <unknown function> + 0x21c9e4 (0x562ba48259e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5201896Z frame #23: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5202108Z frame #24: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5202113Z 
2025-07-17T11:07:25.5202271Z SIGABRT(6), PID: 1777270, Thread 1777798: 
2025-07-17T11:07:25.5202740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5203195Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fe7301da08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5203411Z frame #2: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5203615Z frame #3: pthread_kill + 0x12c (0x7fe821ae69fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5203779Z frame #4: raise + 0x16 (0x7fe821a92476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5203944Z frame #5: abort + 0xd3 (0x7fe821a787f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5204178Z frame #6: <unknown function> + 0x972cc (0x7fe69dd632cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5204410Z frame #7: <unknown function> + 0x95f47 (0x7fe69dd61f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5204631Z frame #8: <unknown function> + 0x34b71 (0x7fe69dd00b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5204847Z frame #9: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5205061Z frame #10: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5205065Z 
2025-07-17T11:07:25.5205164Z SIGABRT(6), PID: 1777267, Thread 1777456: 
2025-07-17T11:07:25.5205701Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5205916Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5206123Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5206329Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5206827Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5207042Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5207255Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5207353Z SIGABRT(6), PID: 1777268, Thread 1781586: 
2025-07-17T11:07:25.5207819Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5208027Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5208187Z frame #2: read + 0x4c (0x7fc44398381c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5208372Z frame #3: _Py_read + 0x4c (0x55e699a23a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5208597Z frame #4: <unknown function> + 0x208c3b (0x55e699a51c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5208819Z frame #5: <unknown function> + 0x14648d (0x55e69998f48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5209065Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55e69999e215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5209352Z frame #7: <unknown function> + 0x208ae7 (0x55e699a51ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5209637Z frame #8: <unknown function> + 0x208472 (0x55e699a51472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5209862Z frame #9: <unknown function> + 0x1da6ee (0x55e699a236ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5210091Z frame #10: <unknown function> + 0x1483bc (0x55e6999913bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5210385Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5210616Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5210856Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55e6999753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5211085Z frame #14: <unknown function> + 0x147734 (0x55e699990734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5211332Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55e699977c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5211562Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5211793Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5212020Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55e6999850ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5212249Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55e6999757b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5212472Z frame #20: <unknown function> + 0x147734 (0x55e699990734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5212688Z frame #21: <unknown function> + 0x21ca82 (0x55e699a65a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5212908Z frame #22: <unknown function> + 0x21c9e4 (0x55e699a659e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5213121Z frame #23: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5213405Z frame #24: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5213409Z 
2025-07-17T11:07:25.5213413Z 
2025-07-17T11:07:25.5213503Z SIGABRT(6), PID: 1777270, Thread 1780310: 
2025-07-17T11:07:25.5213976Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5214187Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5214359Z frame #2: __poll + 0x4f (0x7fe821b68bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5214767Z frame #3: <unknown function> + 0x489f (0x7fe82167489f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.5214998Z frame #4: <unknown function> + 0x1483bc (0x55eb0634f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5215244Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5215478Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5215713Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5215994Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5216238Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55eb06337a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5216468Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5216714Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55eb06335c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5217018Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5217254Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5217555Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5217791Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5218013Z frame #16: <unknown function> + 0x147734 (0x55eb0634e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5218308Z frame #17: <unknown function> + 0x21ca82 (0x55eb06423a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5218532Z frame #18: <unknown function> + 0x21c9e4 (0x55eb064239e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5218745Z frame #19: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5218960Z frame #20: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5218966Z 
2025-07-17T11:07:25.5219057Z SIGABRT(6), PID: 1777267, Thread 1777458: 
2025-07-17T11:07:25.5219539Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5219753Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5219964Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5220169Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5220667Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5220878Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5221095Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5221197Z 
2025-07-17T11:07:25.5221291Z SIGABRT(6), PID: 1777267, Thread 1777461: 
2025-07-17T11:07:25.5221763Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5221971Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5222185Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5222382Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5222881Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5223089Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5223305Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5223313Z 
2025-07-17T11:07:25.5223402Z SIGABRT(6), PID: 1777267, Thread 1777464: 
2025-07-17T11:07:25.5223873Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5224078Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5224288Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5224483Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5224979Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5225257Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5225537Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5225633Z SIGABRT(6), PID: 1777270, Thread 1781250: 
2025-07-17T11:07:25.5226102Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5226379Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5226582Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5226810Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fe821ae42dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5227252Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fe75bf22048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5227679Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fe75bf24bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5227920Z frame #6: <unknown function> + 0xdc253 (0x7fe72e446253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5228132Z frame #7: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5228338Z frame #8: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5228344Z 
2025-07-17T11:07:25.5228348Z 
2025-07-17T11:07:25.5228442Z SIGABRT(6), PID: 1777267, Thread 1777467: 
2025-07-17T11:07:25.5228904Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5229113Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5229322Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5229595Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5230090Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5230300Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5230509Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5230513Z 
2025-07-17T11:07:25.5230606Z SIGABRT(6), PID: 1777269, Thread 1781611: 
2025-07-17T11:07:25.5231069Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5231280Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5231487Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5231689Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5232067Z frame #4: <unknown function> + 0x5a02023 (0x7fb7a74c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5232694Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb7a74c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5233294Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb7a74c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5233762Z frame #7: <unknown function> + 0x79d655 (0x7fb7b89a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5234062Z frame #8: <unknown function> + 0xdc253 (0x7fb76f9fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5234285Z frame #9: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5234501Z frame #10: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5234507Z 
2025-07-17T11:07:25.5234664Z SIGABRT(6), PID: 1777270, Thread 1781251: 
2025-07-17T11:07:25.5235144Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5235352Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5235564Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5235790Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fe821ae42dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5236275Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fe75bf1c741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5236506Z frame #5: <unknown function> + 0xdc253 (0x7fe72e446253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5236719Z frame #6: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5236930Z frame #7: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5236934Z 
2025-07-17T11:07:25.5237029Z SIGABRT(6), PID: 1777268, Thread 1781643: 
2025-07-17T11:07:25.5237502Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5237719Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5237994Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5238198Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5238572Z frame #4: <unknown function> + 0x5a02023 (0x7fc387d30023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5239146Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc387d341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5239744Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc387d2b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5240129Z frame #7: <unknown function> + 0x79d655 (0x7fc39921b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5240362Z frame #8: <unknown function> + 0xdc253 (0x7fc350266253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5240572Z frame #9: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5240785Z frame #10: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5240790Z 
2025-07-17T11:07:25.5240887Z SIGABRT(6), PID: 1777267, Thread 1777470: 
2025-07-17T11:07:25.5241360Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5241569Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5241771Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5242043Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5242618Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5242835Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5243046Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5243107Z 
2025-07-17T11:07:25.5243204Z SIGABRT(6), PID: 1777270, Thread 1781254: 
2025-07-17T11:07:25.5243670Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5243882Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5244057Z frame #2: ioctl + 0x3f (0x7fe821b6a94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5244301Z frame #3: <unknown function> + 0x13eef0 (0x7fe69de0aef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5244534Z frame #4: <unknown function> + 0x1378c4 (0x7fe69de038c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5244766Z frame #5: <unknown function> + 0x95b2b (0x7fe69dd61b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5244993Z frame #6: <unknown function> + 0x34b71 (0x7fe69dd00b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5245210Z frame #7: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5245423Z frame #8: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5245427Z 
2025-07-17T11:07:25.5245518Z SIGABRT(6), PID: 1777267, Thread 1777473: 
2025-07-17T11:07:25.5245994Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5246278Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5246483Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5246687Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5247192Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5247406Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5247619Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5247623Z 
2025-07-17T11:07:25.5247713Z SIGABRT(6), PID: 1777267, Thread 1777476: 
2025-07-17T11:07:25.5248180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5248392Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5248599Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5248800Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5249304Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5249514Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5249726Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5249730Z 
2025-07-17T11:07:25.5249902Z SIGABRT(6), PID: 1777270, Thread 1781541: 
2025-07-17T11:07:25.5250437Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5250650Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5250822Z frame #2: __poll + 0x4f (0x7fe821b68bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5251039Z frame #3: <unknown function> + 0xd4540 (0x7fe6e5f7a540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5251332Z frame #4: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5251541Z frame #5: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5251546Z 
2025-07-17T11:07:25.5251642Z SIGABRT(6), PID: 1777270, Thread 1781549: 
2025-07-17T11:07:25.5252108Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5252325Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5252488Z frame #2: __poll + 0x4f (0x7fe821b68bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5252711Z frame #3: <unknown function> + 0xd65a2 (0x7fe6e5f7c5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5252917Z frame #4: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5253129Z frame #5: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5253133Z 
2025-07-17T11:07:25.5253220Z SIGABRT(6), PID: 1777267, Thread 1777479: 
2025-07-17T11:07:25.5253690Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5253898Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5254172Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5254368Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5254865Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5255070Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5255290Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5255295Z 
2025-07-17T11:07:25.5255383Z SIGABRT(6), PID: 1777270, Thread 1781557: 
2025-07-17T11:07:25.5255855Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5256107Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5256319Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5256518Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5256741Z frame #4: <unknown function> + 0xd1e7f (0x7fe6e5f77e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5256947Z frame #5: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5257158Z frame #6: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5257162Z 
2025-07-17T11:07:25.5257249Z SIGABRT(6), PID: 1777267, Thread 1777482: 
2025-07-17T11:07:25.5257733Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5258020Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5258305Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5258505Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5259002Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5259283Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5259497Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5259503Z 
2025-07-17T11:07:25.5259593Z SIGABRT(6), PID: 1777270, Thread 1781569: 
2025-07-17T11:07:25.5260059Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5260271Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5260475Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5260684Z frame #3: <unknown function> + 0x9ca53 (0x7fe821aeca53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5260939Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55eb0631bbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5261171Z frame #5: <unknown function> + 0x1d314f (0x55eb063da14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5261394Z frame #6: <unknown function> + 0x1d2f6b (0x55eb063d9f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5261617Z frame #7: <unknown function> + 0x6df36 (0x55eb06274f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5261863Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5262092Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5262416Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5262645Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5262880Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5263106Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5263340Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5263565Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5263795Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5264019Z frame #17: <unknown function> + 0x147734 (0x55eb0634e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5264241Z frame #18: <unknown function> + 0x21ca82 (0x55eb06423a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5264463Z frame #19: <unknown function> + 0x21c9e4 (0x55eb064239e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5264680Z frame #20: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5264894Z frame #21: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5264899Z 
2025-07-17T11:07:25.5264988Z SIGABRT(6), PID: 1777269, Thread 1781612: 
2025-07-17T11:07:25.5265467Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5265743Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5265958Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5266214Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5266596Z frame #4: <unknown function> + 0x5a02023 (0x7fb7a74c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5267170Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb7a74c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5267834Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb7a74c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5268218Z frame #7: <unknown function> + 0x79d655 (0x7fb7b89a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5268454Z frame #8: <unknown function> + 0xdc253 (0x7fb76f9fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5268668Z frame #9: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5268886Z frame #10: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5268890Z 
2025-07-17T11:07:25.5268978Z SIGABRT(6), PID: 1777267, Thread 1777485: 
2025-07-17T11:07:25.5269449Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5269657Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5269866Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5270066Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5270641Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5270850Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5271062Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5271067Z 
2025-07-17T11:07:25.5271156Z SIGABRT(6), PID: 1777268, Thread 1781644: 
2025-07-17T11:07:25.5271630Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5271836Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5272046Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5272241Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5272624Z frame #4: <unknown function> + 0x5a02023 (0x7fc387d30023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5273183Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc387d341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5273781Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc387d2b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5274165Z frame #7: <unknown function> + 0x79d655 (0x7fc39921b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5274455Z frame #8: <unknown function> + 0xdc253 (0x7fc350266253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5274732Z frame #9: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5274950Z frame #10: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5274954Z 
2025-07-17T11:07:25.5275045Z SIGABRT(6), PID: 1777267, Thread 1777488: 
2025-07-17T11:07:25.5275512Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5275784Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5275989Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5276192Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5276699Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5276914Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5277123Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5277127Z 
2025-07-17T11:07:25.5277219Z SIGABRT(6), PID: 1777270, Thread 1781585: 
2025-07-17T11:07:25.5277697Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5277906Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5278072Z frame #2: read + 0x4c (0x7fe821b6481c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5278256Z frame #3: _Py_read + 0x4c (0x55eb063e1a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5278484Z frame #4: <unknown function> + 0x208c3b (0x55eb0640fc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5278923Z frame #5: <unknown function> + 0x14648d (0x55eb0634d48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5279171Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55eb0635c215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5279397Z frame #7: <unknown function> + 0x208ae7 (0x55eb0640fae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5279612Z frame #8: <unknown function> + 0x208472 (0x55eb0640f472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5279834Z frame #9: <unknown function> + 0x1da6ee (0x55eb063e16ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5280060Z frame #10: <unknown function> + 0x1483bc (0x55eb0634f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5280303Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5280534Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5280780Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55eb063333bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5281001Z frame #14: <unknown function> + 0x147734 (0x55eb0634e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5281250Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55eb06335c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5281476Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5281715Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5281934Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55eb063430ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5282171Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55eb063337b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5282457Z frame #20: <unknown function> + 0x147734 (0x55eb0634e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5282764Z frame #21: <unknown function> + 0x21ca82 (0x55eb06423a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5282985Z frame #22: <unknown function> + 0x21c9e4 (0x55eb064239e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5283206Z frame #23: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5283418Z frame #24: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5283480Z 
2025-07-17T11:07:25.5283576Z SIGABRT(6), PID: 1777267, Thread 1777491: 
2025-07-17T11:07:25.5284046Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5284260Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5284468Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5284673Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5285171Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5285383Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5285596Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5285600Z 
2025-07-17T11:07:25.5285699Z SIGABRT(6), PID: 1777267, Thread 1777494: 
2025-07-17T11:07:25.5286168Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5286383Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5286661Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5286862Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5287359Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5287570Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5287783Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5287787Z 
2025-07-17T11:07:25.5287878Z SIGABRT(6), PID: 1777267, Thread 1777497: 
2025-07-17T11:07:25.5288341Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5288554Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5288761Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5288960Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5289459Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5289670Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5289881Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5289888Z 
2025-07-17T11:07:25.5289975Z SIGABRT(6), PID: 1777267, Thread 1777501: 
2025-07-17T11:07:25.5290507Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5290781Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5290990Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5291186Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5291685Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5291956Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5292171Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5292176Z 
2025-07-17T11:07:25.5292263Z SIGABRT(6), PID: 1777267, Thread 1777505: 
2025-07-17T11:07:25.5292736Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5292945Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5293153Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5293350Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5293848Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5294056Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5294270Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5294274Z 
2025-07-17T11:07:25.5294364Z SIGABRT(6), PID: 1777269, Thread 1781613: 
2025-07-17T11:07:25.5294834Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5295106Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5295317Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5295512Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5295891Z frame #4: <unknown function> + 0x5a02023 (0x7fb7a74c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5296520Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb7a74c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5297116Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb7a74c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5297499Z frame #7: <unknown function> + 0x79d655 (0x7fb7b89a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5297734Z frame #8: <unknown function> + 0xdc253 (0x7fb76f9fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5297943Z frame #9: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5298168Z frame #10: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5298172Z 
2025-07-17T11:07:25.5298262Z SIGABRT(6), PID: 1777268, Thread 1781645: 
2025-07-17T11:07:25.5298814Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5299022Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5299299Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5299500Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5299872Z frame #4: <unknown function> + 0x5a02023 (0x7fc387d30023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5300518Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc387d341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5301113Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc387d2b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5301497Z frame #7: <unknown function> + 0x79d655 (0x7fc39921b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5301726Z frame #8: <unknown function> + 0xdc253 (0x7fc350266253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5301942Z frame #9: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5302159Z frame #10: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5302165Z 
2025-07-17T11:07:25.5302265Z SIGABRT(6), PID: 1777267, Thread 1777508: 
2025-07-17T11:07:25.5302735Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5302950Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5303156Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5303443Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5303952Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5304161Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5304374Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5304378Z 
2025-07-17T11:07:25.5304471Z SIGABRT(6), PID: 1777270, Thread 1781631: 
2025-07-17T11:07:25.5304943Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5305157Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5305362Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5305568Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5305939Z frame #4: <unknown function> + 0x5a02023 (0x7fe765f10023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5306506Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe765f141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5307101Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe765f0b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5307553Z frame #7: <unknown function> + 0x79d655 (0x7fe7773fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5307857Z frame #8: <unknown function> + 0xdc253 (0x7fe72e446253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5308076Z frame #9: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5308293Z frame #10: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5308302Z 
2025-07-17T11:07:25.5308395Z SIGABRT(6), PID: 1777267, Thread 1777512: 
2025-07-17T11:07:25.5308932Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5309142Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5309348Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5309553Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5310061Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5310270Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5310482Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5310488Z 
2025-07-17T11:07:25.5310577Z SIGABRT(6), PID: 1777267, Thread 1777516: 
2025-07-17T11:07:25.5311053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5311258Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5311469Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5311667Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5312242Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5312452Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5312667Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5312673Z 
2025-07-17T11:07:25.5312762Z SIGABRT(6), PID: 1777267, Thread 1777520: 
2025-07-17T11:07:25.5313229Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5313440Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5313655Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5313855Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5314354Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5314562Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5314779Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5314783Z 
2025-07-17T11:07:25.5314877Z SIGABRT(6), PID: 1777267, Thread 1777524: 
2025-07-17T11:07:25.5315342Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5315635Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5315904Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5316107Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5316612Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5316820Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5317095Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5317100Z 
2025-07-17T11:07:25.5317188Z SIGABRT(6), PID: 1777267, Thread 1777528: 
2025-07-17T11:07:25.5317667Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5317881Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5318097Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5318296Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5318795Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5319006Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5319225Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5319229Z 
2025-07-17T11:07:25.5319321Z SIGABRT(6), PID: 1777267, Thread 1777532: 
2025-07-17T11:07:25.5319796Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5320077Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5320285Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5320484Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5320980Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5321192Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5321401Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5321406Z 
2025-07-17T11:07:25.5321502Z SIGABRT(6), PID: 1777269, Thread 1781614: 
2025-07-17T11:07:25.5321974Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5322194Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5322398Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5322600Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5322977Z frame #4: <unknown function> + 0x5a02023 (0x7fb7a74c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5323553Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb7a74c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5324210Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb7a74c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5324666Z frame #7: <unknown function> + 0x79d655 (0x7fb7b89a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5324906Z frame #8: <unknown function> + 0xdc253 (0x7fb76f9fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5325121Z frame #9: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5325400Z frame #10: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5325404Z 
2025-07-17T11:07:25.5325504Z SIGABRT(6), PID: 1777267, Thread 1777536: 
2025-07-17T11:07:25.5325975Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5326189Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5326399Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5326602Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5327102Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5327320Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5327531Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5327538Z 
2025-07-17T11:07:25.5327635Z SIGABRT(6), PID: 1777270, Thread 1781634: 
2025-07-17T11:07:25.5328109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5328321Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5328619Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5328818Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5329191Z frame #4: <unknown function> + 0x5a02023 (0x7fe765f10023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5329771Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe765f141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5330371Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe765f0b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5330751Z frame #7: <unknown function> + 0x79d655 (0x7fe7773fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5330994Z frame #8: <unknown function> + 0xdc253 (0x7fe72e446253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5331202Z frame #9: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5331419Z frame #10: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5331425Z 
2025-07-17T11:07:25.5331518Z SIGABRT(6), PID: 1777268, Thread 1781646: 
2025-07-17T11:07:25.5331992Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5332198Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5332474Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5332737Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5333116Z frame #4: <unknown function> + 0x5a02023 (0x7fc387d30023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5333681Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc387d341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5334336Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc387d2b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5334716Z frame #7: <unknown function> + 0x79d655 (0x7fc39921b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5334952Z frame #8: <unknown function> + 0xdc253 (0x7fc350266253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5335165Z frame #9: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5335384Z frame #10: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5335388Z 
2025-07-17T11:07:25.5335481Z SIGABRT(6), PID: 1777267, Thread 1777540: 
2025-07-17T11:07:25.5336012Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5336225Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5336439Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5336639Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5337146Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5337439Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5337655Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5337660Z 
2025-07-17T11:07:25.5337750Z SIGABRT(6), PID: 1777267, Thread 1777544: 
2025-07-17T11:07:25.5338226Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5338433Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5338641Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5338845Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5339354Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5339562Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5339778Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5339783Z 
2025-07-17T11:07:25.5339880Z SIGABRT(6), PID: 1777267, Thread 1777548: 
2025-07-17T11:07:25.5340349Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5340560Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5340842Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5341051Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5341617Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5341833Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5342043Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5342115Z 
2025-07-17T11:07:25.5342217Z SIGABRT(6), PID: 1777267, Thread 1777552: 
2025-07-17T11:07:25.5342681Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5342894Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5343104Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5343314Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5343811Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5344028Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5344240Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5344244Z 
2025-07-17T11:07:25.5344340Z SIGABRT(6), PID: 1777267, Thread 1777556: 
2025-07-17T11:07:25.5344804Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5345018Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5345224Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5345516Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5346015Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5346228Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5346440Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5346444Z 
2025-07-17T11:07:25.5346543Z SIGABRT(6), PID: 1777268, Thread 1781647: 
2025-07-17T11:07:25.5347013Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5347230Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5347434Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5347639Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5348014Z frame #4: <unknown function> + 0x5a02023 (0x7fc387d30023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5348586Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc387d341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5349179Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc387d2b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5349633Z frame #7: <unknown function> + 0x79d655 (0x7fc39921b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5349936Z frame #8: <unknown function> + 0xdc253 (0x7fc350266253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5350150Z frame #9: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5350372Z frame #10: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5350435Z 
2025-07-17T11:07:25.5350528Z SIGABRT(6), PID: 1777270, Thread 1781635: 
2025-07-17T11:07:25.5351007Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5351219Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5351432Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5351631Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5352010Z frame #4: <unknown function> + 0x5a02023 (0x7fe765f10023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5352574Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe765f141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5353173Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe765f0b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5353551Z frame #7: <unknown function> + 0x79d655 (0x7fe7773fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5353789Z frame #8: <unknown function> + 0xdc253 (0x7fe72e446253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5354067Z frame #9: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5354292Z frame #10: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5354296Z 
2025-07-17T11:07:25.5354388Z SIGABRT(6), PID: 1777267, Thread 1777559: 
2025-07-17T11:07:25.5354864Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5355073Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5355288Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5355486Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5355998Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5356208Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5356425Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5356429Z 
2025-07-17T11:07:25.5356520Z SIGABRT(6), PID: 1777269, Thread 1781615: 
2025-07-17T11:07:25.5356993Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5357201Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5357411Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5357677Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5358116Z frame #4: <unknown function> + 0x5a02023 (0x7fb7a74c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5358688Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb7a74c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5359290Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb7a74c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5359738Z frame #7: <unknown function> + 0x79d655 (0x7fb7b89a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5359969Z frame #8: <unknown function> + 0xdc253 (0x7fb76f9fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5360185Z frame #9: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5360404Z frame #10: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5360408Z 
2025-07-17T11:07:25.5360503Z SIGABRT(6), PID: 1777267, Thread 1777561: 
2025-07-17T11:07:25.5360971Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5361188Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5361396Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5361605Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5362110Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5362322Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5362602Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5362606Z 
2025-07-17T11:07:25.5362704Z SIGABRT(6), PID: 1777267, Thread 1777564: 
2025-07-17T11:07:25.5363170Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5363383Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5363588Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5363790Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5364296Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5364511Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5364726Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5364730Z 
2025-07-17T11:07:25.5364830Z SIGABRT(6), PID: 1777267, Thread 1777567: 
2025-07-17T11:07:25.5365294Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5365508Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5365712Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5365918Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5366533Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5366756Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5366970Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5366974Z 
2025-07-17T11:07:25.5367076Z SIGABRT(6), PID: 1777267, Thread 1777570: 
2025-07-17T11:07:25.5367614Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5367828Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5368042Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5368251Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5368750Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5368965Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5369176Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5369186Z 
2025-07-17T11:07:25.5369279Z SIGABRT(6), PID: 1777267, Thread 1777573: 
2025-07-17T11:07:25.5369744Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5369953Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5370161Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5370360Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5370933Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5371142Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5371359Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5371365Z 
2025-07-17T11:07:25.5371455Z SIGABRT(6), PID: 1777268, Thread 1781648: 
2025-07-17T11:07:25.5371934Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5372140Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5372353Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5372554Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5372934Z frame #4: <unknown function> + 0x5a02023 (0x7fc387d30023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5373492Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc387d341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5374091Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc387d2b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5374487Z frame #7: <unknown function> + 0x79d655 (0x7fc39921b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5374786Z frame #8: <unknown function> + 0xdc253 (0x7fc350266253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5375060Z frame #9: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5375285Z frame #10: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5375289Z 
2025-07-17T11:07:25.5375379Z SIGABRT(6), PID: 1777267, Thread 1777576: 
2025-07-17T11:07:25.5375864Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5376209Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5376425Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5376630Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5377154Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5377363Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5377583Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5377587Z 
2025-07-17T11:07:25.5377679Z SIGABRT(6), PID: 1777270, Thread 1781637: 
2025-07-17T11:07:25.5378165Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5378373Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5378581Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5378779Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5379248Z frame #4: <unknown function> + 0x5a02023 (0x7fe765f10023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5379813Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe765f141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5380418Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe765f0b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5380809Z frame #7: <unknown function> + 0x79d655 (0x7fe7773fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5381041Z frame #8: <unknown function> + 0xdc253 (0x7fe72e446253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5381256Z frame #9: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5381477Z frame #10: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5381481Z 
2025-07-17T11:07:25.5381577Z SIGABRT(6), PID: 1777269, Thread 1781616: 
2025-07-17T11:07:25.5382056Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5382272Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5382475Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5382678Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5383124Z frame #4: <unknown function> + 0x5a02023 (0x7fb7a74c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5383767Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb7a74c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5384371Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb7a74c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5384826Z frame #7: <unknown function> + 0x79d655 (0x7fb7b89a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5385054Z frame #8: <unknown function> + 0xdc253 (0x7fb76f9fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5385270Z frame #9: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5385489Z frame #10: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5385493Z 
2025-07-17T11:07:25.5385592Z SIGABRT(6), PID: 1777267, Thread 1777579: 
2025-07-17T11:07:25.5386064Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5386281Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5386488Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5386695Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5387201Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5387417Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5387632Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5387700Z 
2025-07-17T11:07:25.5387799Z SIGABRT(6), PID: 1777267, Thread 1777582: 
2025-07-17T11:07:25.5388272Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5388490Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5388697Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5388899Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5389401Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5389618Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5389831Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5389841Z 
2025-07-17T11:07:25.5389933Z SIGABRT(6), PID: 1777267, Thread 1777585: 
2025-07-17T11:07:25.5390404Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5390611Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5390829Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5391030Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5391606Z frame #4: <unknown function> + 0x3475ab (0x7f88a7aca5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5391824Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5392103Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5392108Z 
2025-07-17T11:07:25.5392207Z SIGABRT(6), PID: 1777267, Thread 1777785: 
2025-07-17T11:07:25.5392684Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5393199Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f87b932b08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5393417Z frame #2: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5393617Z frame #3: pthread_kill + 0x12c (0x7f88aac349fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5393794Z frame #4: raise + 0x16 (0x7f88aabe0476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5393957Z frame #5: abort + 0xd3 (0x7f88aabc67f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5394204Z frame #6: <unknown function> + 0x972cc (0x7f8726eb42cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5394437Z frame #7: <unknown function> + 0x95f47 (0x7f8726eb2f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5394669Z frame #8: <unknown function> + 0x34b71 (0x7f8726e51b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5394881Z frame #9: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5395104Z frame #10: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5395108Z 
2025-07-17T11:07:25.5395203Z SIGABRT(6), PID: 1777267, Thread 1780113: 
2025-07-17T11:07:25.5395682Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5395961Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5396136Z frame #2: __poll + 0x4f (0x7f88aacb6bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5396545Z frame #3: <unknown function> + 0x489f (0x7f88aa7c289f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.5396781Z frame #4: <unknown function> + 0x1483bc (0x55d3af82a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5397030Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5397263Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5397500Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5397730Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5397983Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55d3af812a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5398223Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5398475Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d3af810c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5398712Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5398953Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5399185Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5399425Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5399722Z frame #16: <unknown function> + 0x147734 (0x55d3af829734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5400009Z frame #17: <unknown function> + 0x21ca82 (0x55d3af8fea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5400248Z frame #18: <unknown function> + 0x21c9e4 (0x55d3af8fe9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5400469Z frame #19: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5400693Z frame #20: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5400758Z 
2025-07-17T11:07:25.5400856Z SIGABRT(6), PID: 1777268, Thread 1781649: 
2025-07-17T11:07:25.5401347Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5401562Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5401783Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5401993Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5402385Z frame #4: <unknown function> + 0x5a02023 (0x7fc387d30023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5402952Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc387d341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5403556Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc387d2b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5403939Z frame #7: <unknown function> + 0x79d655 (0x7fc39921b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5404183Z frame #8: <unknown function> + 0xdc253 (0x7fc350266253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5404465Z frame #9: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5404689Z frame #10: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5404694Z 
2025-07-17T11:07:25.5404790Z SIGABRT(6), PID: 1777270, Thread 1781639: 
2025-07-17T11:07:25.5405269Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5405486Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5405691Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5405900Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5406281Z frame #4: <unknown function> + 0x5a02023 (0x7fe765f10023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5406856Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe765f141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5407447Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe765f0b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5407841Z frame #7: <unknown function> + 0x79d655 (0x7fe7773fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5408073Z frame #8: <unknown function> + 0xdc253 (0x7fe72e446253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5408291Z frame #9: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5408571Z frame #10: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5408579Z 
2025-07-17T11:07:25.5408766Z SIGABRT(6), PID: 1777269, Thread 1781617: 
2025-07-17T11:07:25.5409247Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5409467Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5409891Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5410098Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5410473Z frame #4: <unknown function> + 0x5a02023 (0x7fb7a74c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5411049Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb7a74c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5411645Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb7a74c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5412034Z frame #7: <unknown function> + 0x79d655 (0x7fb7b89a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5412269Z frame #8: <unknown function> + 0xdc253 (0x7fb76f9fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5412482Z frame #9: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5412700Z frame #10: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5412703Z 
2025-07-17T11:07:25.5412805Z SIGABRT(6), PID: 1777271, Thread 1777271: 
2025-07-17T11:07:25.5413281Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5413571Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5413769Z frame #2: pthread_kill + 0x12c (0x7f83309829fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5413945Z frame #3: raise + 0x16 (0x7f833092e476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5414323Z frame #4: <unknown function> + 0x4feaa4d (0x7f818d50aa4d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/_C/libtriton.so)
2025-07-17T11:07:25.5414537Z frame #5: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5414767Z frame #6: <unknown function> + 0x77445 (0x7f81acbe0445 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5415009Z frame #7: <unknown function> + 0x76c8e (0x7f81acbdfc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5415238Z frame #8: <unknown function> + 0x6ad69 (0x7f81acbd3d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5415465Z frame #9: <unknown function> + 0x3f54db (0x7f823d8384db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5415683Z frame #10: <unknown function> + 0x3e11d8 (0x7f823d8241d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5415951Z frame #11: <unknown function> + 0xb2f86 (0x7f823d4f5f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5416166Z frame #12: <unknown function> + 0x8e501 (0x7f823d4d1501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5416561Z frame #13: c10::hip::device_synchronize() + 0x5a (0x7f82682d87ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.5416948Z frame #14: <unknown function> + 0xd307d9 (0x7f828682b7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5417259Z frame #15: <unknown function> + 0x139902 (0x561664218902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5417598Z frame #16: _PyEval_EvalFrameDefault + 0x49b5 (0x56166420fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5417843Z frame #17: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5418088Z frame #18: _PyEval_EvalFrameDefault + 0x49b5 (0x56166420fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5418420Z frame #19: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5418661Z frame #20: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5418894Z frame #21: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5419133Z frame #22: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5419361Z frame #23: <unknown function> + 0x1474b2 (0x5616642264b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5419568Z frame #24: PyObject_Call + 0xbc (0x561664226e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5419798Z frame #25: <unknown function> + 0x1ff69d (0x5616642de69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5420021Z frame #26: _PyObject_MakeTpCall + 0x2d3 (0x561664214023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5420272Z frame #27: _PyEval_EvalFrameDefault + 0x4c16 (0x56166420fcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5420500Z frame #28: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5420752Z frame #29: _PyEval_EvalFrameDefault + 0x30c (0x56166420b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5420977Z frame #30: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5421220Z frame #31: _PyEval_EvalFrameDefault + 0x30c (0x56166420b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5421533Z frame #32: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5421786Z frame #33: _PyEval_EvalFrameDefault + 0x49b5 (0x56166420fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5422044Z frame #34: _PyObject_FastCallDictTstate + 0xd0 (0x5616642134b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5422272Z frame #35: _PyObject_Call_Prepend + 0x69 (0x561664224ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5422493Z frame #36: <unknown function> + 0x205949 (0x5616642e4949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5422720Z frame #37: _PyObject_MakeTpCall + 0x2d3 (0x561664214023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5422967Z frame #38: _PyEval_EvalFrameDefault + 0x4c16 (0x56166420fcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5423204Z frame #39: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5423446Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x56166420b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5423674Z frame #41: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5423911Z frame #42: _PyEval_EvalFrameDefault + 0x30c (0x56166420b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5424138Z frame #43: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5424379Z frame #44: _PyEval_EvalFrameDefault + 0x30c (0x56166420b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5424611Z frame #45: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5424853Z frame #46: _PyEval_EvalFrameDefault + 0x1340 (0x56166420c3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5425148Z frame #47: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5425607Z frame #48: <unknown function> + 0x7ab3f7 (0x7f82862a63f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5425837Z frame #49: <unknown function> + 0x13bc88 (0x56166421ac88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5426044Z frame #50: PyObject_Call + 0x20f (0x561664226f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5426291Z frame #51: _PyEval_EvalFrameDefault + 0x58dd (0x56166421098d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5426574Z frame #52: <unknown function> + 0x147608 (0x561664226608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5426826Z frame #53: _PyEval_EvalFrameDefault + 0x2ba5 (0x56166420dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5427049Z frame #54: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5427291Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x56166420b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5427523Z frame #56: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5427773Z frame #57: _PyEval_EvalFrameDefault + 0x1340 (0x56166420c3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5428004Z frame #58: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5428250Z frame #59: _PyEval_EvalFrameDefault + 0x30c (0x56166420b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5428476Z frame #60: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5428724Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x56166420dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5428951Z frame #62: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5428955Z 
2025-07-17T11:07:25.5429057Z SIGABRT(6), PID: 1777265, Thread 1777265: 
2025-07-17T11:07:25.5429616Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5429844Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5430041Z frame #2: pthread_kill + 0x12c (0x7f8babc789fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5430212Z frame #3: raise + 0x16 (0x7f8babc24476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5430587Z frame #4: <unknown function> + 0x4feaa4d (0x7f8a087fca4d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/_C/libtriton.so)
2025-07-17T11:07:25.5430812Z frame #5: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5431044Z frame #6: <unknown function> + 0x77435 (0x7f8a27ed2435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5431278Z frame #7: <unknown function> + 0x76c8e (0x7f8a27ed1c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5431508Z frame #8: <unknown function> + 0x6ad69 (0x7f8a27ec5d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5431732Z frame #9: <unknown function> + 0x3f54db (0x7f8ab8b2a4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5431952Z frame #10: <unknown function> + 0x3e11d8 (0x7f8ab8b161d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5432176Z frame #11: <unknown function> + 0xb2f86 (0x7f8ab87e7f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5432384Z frame #12: <unknown function> + 0x8e501 (0x7f8ab87c3501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5432779Z frame #13: c10::hip::device_synchronize() + 0x5a (0x7f8ae35ca7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.5433236Z frame #14: <unknown function> + 0xd307d9 (0x7f8b01b1d7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5433470Z frame #15: <unknown function> + 0x139902 (0x5587304ed902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5433777Z frame #16: _PyEval_EvalFrameDefault + 0x49b5 (0x5587304e4a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5434016Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5434258Z frame #18: _PyEval_EvalFrameDefault + 0x49b5 (0x5587304e4a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5434551Z frame #19: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5434792Z frame #20: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5435021Z frame #21: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5435255Z frame #22: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5435488Z frame #23: <unknown function> + 0x1474b2 (0x5587304fb4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5435699Z frame #24: PyObject_Call + 0xbc (0x5587304fbe4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5435930Z frame #25: <unknown function> + 0x1ff69d (0x5587305b369d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5436150Z frame #26: _PyObject_MakeTpCall + 0x2d3 (0x5587304e9023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5436411Z frame #27: _PyEval_EvalFrameDefault + 0x4c16 (0x5587304e4cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5436636Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5436880Z frame #29: _PyEval_EvalFrameDefault + 0x30c (0x5587304e03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5437102Z frame #30: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5437349Z frame #31: _PyEval_EvalFrameDefault + 0x30c (0x5587304e03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5437647Z frame #32: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5437904Z frame #33: _PyEval_EvalFrameDefault + 0x49b5 (0x5587304e4a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5438160Z frame #34: _PyObject_FastCallDictTstate + 0xd0 (0x5587304e84b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5438387Z frame #35: _PyObject_Call_Prepend + 0x69 (0x5587304f9ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5438608Z frame #36: <unknown function> + 0x205949 (0x5587305b9949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5438837Z frame #37: _PyObject_MakeTpCall + 0x2d3 (0x5587304e9023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5439081Z frame #38: _PyEval_EvalFrameDefault + 0x4c16 (0x5587304e4cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5439318Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5439556Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5587304e03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5439784Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5440017Z frame #42: _PyEval_EvalFrameDefault + 0x30c (0x5587304e03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5440246Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5440486Z frame #44: _PyEval_EvalFrameDefault + 0x30c (0x5587304e03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5440720Z frame #45: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5440958Z frame #46: _PyEval_EvalFrameDefault + 0x1340 (0x5587304e13f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5441282Z frame #47: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5441735Z frame #48: <unknown function> + 0x7ab3f7 (0x7f8b015983f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5441968Z frame #49: <unknown function> + 0x13bc88 (0x5587304efc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5442173Z frame #50: PyObject_Call + 0x20f (0x5587304fbf9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5442487Z frame #51: _PyEval_EvalFrameDefault + 0x58dd (0x5587304e598d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5442709Z frame #52: <unknown function> + 0x147608 (0x5587304fb608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5442956Z frame #53: _PyEval_EvalFrameDefault + 0x2ba5 (0x5587304e2c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5443184Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5443433Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5587304e03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5443659Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5443910Z frame #57: _PyEval_EvalFrameDefault + 0x1340 (0x5587304e13f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5444133Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5444376Z frame #59: _PyEval_EvalFrameDefault + 0x30c (0x5587304e03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5444600Z frame #60: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5444848Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5587304e2c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5445071Z frame #62: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5445081Z 
2025-07-17T11:07:25.5445180Z SIGABRT(6), PID: 1777267, Thread 1781108: 
2025-07-17T11:07:25.5445748Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5445969Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5446189Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5446421Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f88aac322dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5446878Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f87e5073048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5447301Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f87e5075bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5447537Z frame #6: <unknown function> + 0xdc253 (0x7f87b7597253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5447751Z frame #7: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5447975Z frame #8: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5447978Z 
2025-07-17T11:07:25.5448076Z SIGABRT(6), PID: 1777271, Thread 1777586: 
2025-07-17T11:07:25.5448561Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5448773Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5448984Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5449186Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5449818Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5450041Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5450264Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5450270Z 
2025-07-17T11:07:25.5450365Z SIGABRT(6), PID: 1777265, Thread 1777273: 
2025-07-17T11:07:25.5450907Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5451116Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5451327Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5451530Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5452045Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5452258Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5452475Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5452481Z 
2025-07-17T11:07:25.5452573Z SIGABRT(6), PID: 1777271, Thread 1777589: 
2025-07-17T11:07:25.5453050Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5453255Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5453471Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5453670Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5454247Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5454460Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5454676Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5454682Z 
2025-07-17T11:07:25.5454776Z SIGABRT(6), PID: 1777267, Thread 1781109: 
2025-07-17T11:07:25.5455254Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5455463Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5455676Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5455972Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f88aac322dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5456455Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f87e506d741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.5456689Z frame #5: <unknown function> + 0xdc253 (0x7f87b7597253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5456907Z frame #6: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5457120Z frame #7: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5457130Z 
2025-07-17T11:07:25.5457221Z SIGABRT(6), PID: 1777265, Thread 1777274: 
2025-07-17T11:07:25.5457776Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5458064Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5458281Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5458513Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5459054Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5459342Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5459566Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5459571Z 
2025-07-17T11:07:25.5459666Z SIGABRT(6), PID: 1777267, Thread 1781110: 
2025-07-17T11:07:25.5460148Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5460361Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5460539Z frame #2: ioctl + 0x3f (0x7f88aacb894f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5460774Z frame #3: <unknown function> + 0x13eef0 (0x7f8726f5bef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5461008Z frame #4: <unknown function> + 0x1378c4 (0x7f8726f548c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5461241Z frame #5: <unknown function> + 0x95b2b (0x7f8726eb2b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5461470Z frame #6: <unknown function> + 0x34b71 (0x7f8726e51b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5461680Z frame #7: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5461899Z frame #8: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5461980Z 
2025-07-17T11:07:25.5462077Z SIGABRT(6), PID: 1777271, Thread 1777593: 
2025-07-17T11:07:25.5462554Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5462763Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5462974Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5463177Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5463677Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5463888Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5464104Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5464112Z 
2025-07-17T11:07:25.5464204Z SIGABRT(6), PID: 1777265, Thread 1777275: 
2025-07-17T11:07:25.5464674Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5464882Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5465099Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5465299Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5465810Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5466084Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5466366Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5466371Z 
2025-07-17T11:07:25.5466464Z SIGABRT(6), PID: 1777268, Thread 1781650: 
2025-07-17T11:07:25.5466945Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc351ff9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5467216Z frame #1: <unknown function> + 0x42520 (0x7fc4438b1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5467429Z frame #2: <unknown function> + 0x91117 (0x7fc443900117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5467629Z frame #3: pthread_cond_wait + 0x211 (0x7fc443902a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5468035Z frame #4: <unknown function> + 0x5a02023 (0x7fc387d30023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5468608Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc387d341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5469214Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc387d2b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5469611Z frame #7: <unknown function> + 0x79d655 (0x7fc39921b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5469847Z frame #8: <unknown function> + 0xdc253 (0x7fc350266253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5470078Z frame #9: <unknown function> + 0x94ac3 (0x7fc443903ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5470298Z frame #10: <unknown function> + 0x126850 (0x7fc443995850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5470303Z 
2025-07-17T11:07:25.5470495Z SIGABRT(6), PID: 1777271, Thread 1777596: 
2025-07-17T11:07:25.5470969Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5471181Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5471385Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5471596Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5472095Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5472312Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5472526Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5472627Z SIGABRT(6), PID: 1777267, Thread 1781546: 
2025-07-17T11:07:25.5473099Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5473317Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5473494Z frame #2: __poll + 0x4f (0x7f88aacb6bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5473717Z frame #3: <unknown function> + 0xd4540 (0x7f876f0cb540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5473926Z frame #4: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5474145Z frame #5: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5474149Z 
2025-07-17T11:07:25.5474152Z 
2025-07-17T11:07:25.5474311Z SIGABRT(6), PID: 1777269, Thread 1781618: 
2025-07-17T11:07:25.5474850Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb77178eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5475067Z frame #1: <unknown function> + 0x42520 (0x7fb863046520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5475282Z frame #2: <unknown function> + 0x91117 (0x7fb863095117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5475481Z frame #3: pthread_cond_wait + 0x211 (0x7fb863097a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5475932Z frame #4: <unknown function> + 0x5a02023 (0x7fb7a74c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5476505Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb7a74c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5477110Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb7a74c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5477499Z frame #7: <unknown function> + 0x79d655 (0x7fb7b89a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5477738Z frame #8: <unknown function> + 0xdc253 (0x7fb76f9fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5477954Z frame #9: <unknown function> + 0x94ac3 (0x7fb863098ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5478176Z frame #10: <unknown function> + 0x126850 (0x7fb86312a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5478267Z SIGABRT(6), PID: 1777265, Thread 1777276: 
2025-07-17T11:07:25.5478749Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5478961Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5479242Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5479445Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5479952Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5480165Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5480381Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5480385Z 
2025-07-17T11:07:25.5480388Z 
2025-07-17T11:07:25.5480497Z SIGABRT(6), PID: 1777267, Thread 1781554: 
2025-07-17T11:07:25.5480971Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5481198Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5481370Z frame #2: __poll + 0x4f (0x7f88aacb6bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5481595Z frame #3: <unknown function> + 0xd65a2 (0x7f876f0cd5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5481803Z frame #4: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5482019Z frame #5: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5482023Z 
2025-07-17T11:07:25.5482116Z SIGABRT(6), PID: 1777271, Thread 1777599: 
2025-07-17T11:07:25.5482594Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5482863Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5483136Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5483339Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5483844Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5484114Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5484326Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5484330Z 
2025-07-17T11:07:25.5484422Z SIGABRT(6), PID: 1777267, Thread 1781561: 
2025-07-17T11:07:25.5484899Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5485112Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5485326Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5485525Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5485747Z frame #4: <unknown function> + 0xd1e7f (0x7f876f0c8e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.5485956Z frame #5: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5486174Z frame #6: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5486178Z 
2025-07-17T11:07:25.5486271Z SIGABRT(6), PID: 1777265, Thread 1777277: 
2025-07-17T11:07:25.5486748Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5487026Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5487240Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5487447Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5487954Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5488167Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5488389Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5488393Z 
2025-07-17T11:07:25.5488485Z SIGABRT(6), PID: 1777267, Thread 1781568: 
2025-07-17T11:07:25.5488960Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5489177Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5489392Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5489601Z frame #3: <unknown function> + 0x9ca53 (0x7f88aac3aa53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5489868Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55d3af7f6bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5490103Z frame #5: <unknown function> + 0x1d314f (0x55d3af8b514f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5490334Z frame #6: <unknown function> + 0x1d2f6b (0x55d3af8b4f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5490557Z frame #7: <unknown function> + 0x6df36 (0x55d3af74ff36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5490869Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5491178Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5491435Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5491669Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5491915Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5492210Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5492452Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5492905Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5493242Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5493556Z frame #17: <unknown function> + 0x147734 (0x55d3af829734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5493824Z frame #18: <unknown function> + 0x21ca82 (0x55d3af8fea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5494108Z frame #19: <unknown function> + 0x21c9e4 (0x55d3af8fe9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5507653Z frame #20: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5507967Z frame #21: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5507974Z 
2025-07-17T11:07:25.5508122Z SIGABRT(6), PID: 1777271, Thread 1777602: 
2025-07-17T11:07:25.5508644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5508885Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5509268Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5509487Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5510009Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5510237Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5510457Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5510463Z 
2025-07-17T11:07:25.5510569Z SIGABRT(6), PID: 1777270, Thread 1781640: 
2025-07-17T11:07:25.5511070Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5511307Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5511525Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5511740Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5512142Z frame #4: <unknown function> + 0x5a02023 (0x7fe765f10023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5512733Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe765f141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5513426Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe765f0b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5513905Z frame #7: <unknown function> + 0x79d655 (0x7fe7773fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5514160Z frame #8: <unknown function> + 0xdc253 (0x7fe72e446253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5514394Z frame #9: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5514624Z frame #10: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5514710Z 
2025-07-17T11:07:25.5514819Z SIGABRT(6), PID: 1777265, Thread 1777278: 
2025-07-17T11:07:25.5515325Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5515548Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5515761Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5515977Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5516497Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5516711Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5516933Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5516937Z 
2025-07-17T11:07:25.5517033Z SIGABRT(6), PID: 1777267, Thread 1781580: 
2025-07-17T11:07:25.5517514Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5517728Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5517901Z frame #2: read + 0x4c (0x7f88aacb281c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5518173Z frame #3: _Py_read + 0x4c (0x55d3af8bca6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5518413Z frame #4: <unknown function> + 0x208c3b (0x55d3af8eac3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5518640Z frame #5: <unknown function> + 0x14648d (0x55d3af82848d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5518899Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55d3af837215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5519127Z frame #7: <unknown function> + 0x208ae7 (0x55d3af8eaae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5519352Z frame #8: <unknown function> + 0x208472 (0x55d3af8ea472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5519576Z frame #9: <unknown function> + 0x1da6ee (0x55d3af8bc6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5519812Z frame #10: <unknown function> + 0x1483bc (0x55d3af82a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5520064Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5520299Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5520540Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55d3af80e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5520767Z frame #14: <unknown function> + 0x147734 (0x55d3af829734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5521019Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d3af810c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5521253Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5521491Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5521786Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55d3af81e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5522092Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55d3af80e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5522332Z frame #20: <unknown function> + 0x147734 (0x55d3af829734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5522557Z frame #21: <unknown function> + 0x21ca82 (0x55d3af8fea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5522785Z frame #22: <unknown function> + 0x21c9e4 (0x55d3af8fe9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5523206Z frame #23: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5523431Z frame #24: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5523436Z 
2025-07-17T11:07:25.5523539Z SIGABRT(6), PID: 1777271, Thread 1777605: 
2025-07-17T11:07:25.5524050Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5524276Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5524488Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5524698Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5525215Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5525426Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5525642Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5525646Z 
2025-07-17T11:07:25.5525743Z SIGABRT(6), PID: 1777265, Thread 1777279: 
2025-07-17T11:07:25.5526228Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5526519Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5526734Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5526939Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5527454Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5527668Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5527883Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5527888Z 
2025-07-17T11:07:25.5527983Z SIGABRT(6), PID: 1777271, Thread 1777608: 
2025-07-17T11:07:25.5528462Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5528671Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5528877Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5529083Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5529582Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5529790Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5530072Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5530077Z 
2025-07-17T11:07:25.5530177Z SIGABRT(6), PID: 1777265, Thread 1777280: 
2025-07-17T11:07:25.5530709Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5530926Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5531141Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5531404Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5531902Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5532117Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5532330Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5532336Z 
2025-07-17T11:07:25.5532438Z SIGABRT(6), PID: 1777271, Thread 1777611: 
2025-07-17T11:07:25.5532904Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5533114Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5533321Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5554137Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5554666Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5554889Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5555216Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5555221Z 
2025-07-17T11:07:25.5555323Z SIGABRT(6), PID: 1777265, Thread 1777281: 
2025-07-17T11:07:25.5555798Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5556012Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5556218Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5556424Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5556932Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5557142Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5557358Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5557362Z 
2025-07-17T11:07:25.5557458Z SIGABRT(6), PID: 1777271, Thread 1777614: 
2025-07-17T11:07:25.5557920Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5558131Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5558333Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5558535Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5559107Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5559422Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5559633Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5559638Z 
2025-07-17T11:07:25.5559738Z SIGABRT(6), PID: 1777265, Thread 1777282: 
2025-07-17T11:07:25.5560214Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5560506Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5560714Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5560918Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5561444Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5561665Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5561878Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5561883Z 
2025-07-17T11:07:25.5561982Z SIGABRT(6), PID: 1777271, Thread 1777617: 
2025-07-17T11:07:25.5562466Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5562678Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5562886Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5563087Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5563662Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5563873Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5564083Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5564089Z 
2025-07-17T11:07:25.5564178Z SIGABRT(6), PID: 1777270, Thread 1781641: 
2025-07-17T11:07:25.5564660Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5564868Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5565075Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5565271Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5565660Z frame #4: <unknown function> + 0x5a02023 (0x7fe765f10023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5566232Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe765f141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5566829Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe765f0b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5567212Z frame #7: <unknown function> + 0x79d655 (0x7fe7773fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5567522Z frame #8: <unknown function> + 0xdc253 (0x7fe72e446253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5567807Z frame #9: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5568035Z frame #10: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5568039Z 
2025-07-17T11:07:25.5568129Z SIGABRT(6), PID: 1777265, Thread 1777283: 
2025-07-17T11:07:25.5568604Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5568876Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5569081Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5569281Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5569784Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5569998Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5570209Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5570214Z 
2025-07-17T11:07:25.5570302Z SIGABRT(6), PID: 1777271, Thread 1777620: 
2025-07-17T11:07:25.5570771Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5570979Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5571181Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5571376Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5571875Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5572150Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5572360Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5572364Z 
2025-07-17T11:07:25.5572454Z SIGABRT(6), PID: 1777267, Thread 1781619: 
2025-07-17T11:07:25.5572924Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5573133Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5573336Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5573538Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5573916Z frame #4: <unknown function> + 0x5a02023 (0x7f87ef061023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5574479Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87ef0651ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5575071Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87ef05c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5575453Z frame #7: <unknown function> + 0x79d655 (0x7f880054c655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5575683Z frame #8: <unknown function> + 0xdc253 (0x7f87b7597253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5576050Z frame #9: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5576355Z frame #10: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5576360Z 
2025-07-17T11:07:25.5576458Z SIGABRT(6), PID: 1777265, Thread 1777284: 
2025-07-17T11:07:25.5576924Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5577208Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5577410Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5577612Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5578110Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5578326Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5578537Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5578541Z 
2025-07-17T11:07:25.5578631Z SIGABRT(6), PID: 1777271, Thread 1777623: 
2025-07-17T11:07:25.5579093Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5579301Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5579502Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5579699Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5580193Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5580482Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5601343Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5601353Z 
2025-07-17T11:07:25.5601526Z SIGABRT(6), PID: 1777265, Thread 1777285: 
2025-07-17T11:07:25.5602352Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5602710Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5603069Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5603414Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5604274Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5604633Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5605003Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5605011Z 
2025-07-17T11:07:25.5605169Z SIGABRT(6), PID: 1777271, Thread 1777626: 
2025-07-17T11:07:25.5605996Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5606356Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5606717Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5607187Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5608159Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5608536Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5608778Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5608847Z 
2025-07-17T11:07:25.5608940Z SIGABRT(6), PID: 1777265, Thread 1777286: 
2025-07-17T11:07:25.5609417Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5609629Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5609839Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5610049Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5610548Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5610761Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5610973Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5610977Z 
2025-07-17T11:07:25.5611071Z SIGABRT(6), PID: 1777271, Thread 1777630: 
2025-07-17T11:07:25.5611536Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5611744Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5611949Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5612226Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5612721Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5612930Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5613141Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5613144Z 
2025-07-17T11:07:25.5613238Z SIGABRT(6), PID: 1777265, Thread 1777287: 
2025-07-17T11:07:25.5613703Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5613917Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5614124Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5614328Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5614822Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5615038Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5615247Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5615251Z 
2025-07-17T11:07:25.5615344Z SIGABRT(6), PID: 1777271, Thread 1777634: 
2025-07-17T11:07:25.5615901Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5616272Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5616497Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5616707Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5617202Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5617496Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5617705Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5617709Z 
2025-07-17T11:07:25.5617807Z SIGABRT(6), PID: 1777270, Thread 1781642: 
2025-07-17T11:07:25.5618276Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe7301d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5618492Z frame #1: <unknown function> + 0x42520 (0x7fe821a92520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5618698Z frame #2: <unknown function> + 0x91117 (0x7fe821ae1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5618901Z frame #3: pthread_cond_wait + 0x211 (0x7fe821ae3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5619277Z frame #4: <unknown function> + 0x5a02023 (0x7fe765f10023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5619845Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe765f141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5620450Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe765f0b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5620920Z frame #7: <unknown function> + 0x79d655 (0x7fe7773fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5621160Z frame #8: <unknown function> + 0xdc253 (0x7fe72e446253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5621370Z frame #9: <unknown function> + 0x94ac3 (0x7fe821ae4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5621590Z frame #10: <unknown function> + 0x126850 (0x7fe821b76850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5621594Z 
2025-07-17T11:07:25.5621685Z SIGABRT(6), PID: 1777265, Thread 1777288: 
2025-07-17T11:07:25.5622163Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5622374Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5622588Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5622787Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5623295Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5623502Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5623720Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5623724Z 
2025-07-17T11:07:25.5623815Z SIGABRT(6), PID: 1777271, Thread 1777638: 
2025-07-17T11:07:25.5624375Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5624586Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5624856Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5625055Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5625554Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5625824Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5626037Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5626041Z 
2025-07-17T11:07:25.5626137Z SIGABRT(6), PID: 1777265, Thread 1777289: 
2025-07-17T11:07:25.5626613Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5626823Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5627032Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5627229Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5627732Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5627941Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5628158Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5628162Z 
2025-07-17T11:07:25.5628252Z SIGABRT(6), PID: 1777271, Thread 1777642: 
2025-07-17T11:07:25.5628727Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5629005Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5629214Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5629411Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5629906Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5630111Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5630323Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5630327Z 
2025-07-17T11:07:25.5630420Z SIGABRT(6), PID: 1777267, Thread 1781620: 
2025-07-17T11:07:25.5630896Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5631108Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5631313Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5631512Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5631889Z frame #4: <unknown function> + 0x5a02023 (0x7f87ef061023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5632454Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87ef0651ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5633195Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87ef05c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5633594Z frame #7: <unknown function> + 0x79d655 (0x7f880054c655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5633829Z frame #8: <unknown function> + 0xdc253 (0x7f87b7597253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5634120Z frame #9: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5634352Z frame #10: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5634357Z 
2025-07-17T11:07:25.5634452Z SIGABRT(6), PID: 1777265, Thread 1777290: 
2025-07-17T11:07:25.5634929Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5635144Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5635353Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5635559Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5636060Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5636274Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5636486Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5636490Z 
2025-07-17T11:07:25.5636586Z SIGABRT(6), PID: 1777271, Thread 1777646: 
2025-07-17T11:07:25.5637050Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5637331Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5637535Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5637737Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5638233Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5638445Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5638654Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5638657Z 
2025-07-17T11:07:25.5638752Z SIGABRT(6), PID: 1777271, Thread 1777651: 
2025-07-17T11:07:25.5639227Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5639439Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5639639Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5639841Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5640333Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5640540Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5640749Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5640757Z 
2025-07-17T11:07:25.5640920Z SIGABRT(6), PID: 1777265, Thread 1777291: 
2025-07-17T11:07:25.5641455Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5641667Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5641877Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5642136Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5642631Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5642839Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5643056Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5643060Z 
2025-07-17T11:07:25.5643152Z SIGABRT(6), PID: 1777271, Thread 1777655: 
2025-07-17T11:07:25.5643620Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5643826Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5644032Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5644228Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5644725Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5644942Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5645156Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5645248Z 
2025-07-17T11:07:25.5645341Z SIGABRT(6), PID: 1777265, Thread 1777292: 
2025-07-17T11:07:25.5645815Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5646023Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5646235Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5646433Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5646929Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5647137Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5647354Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5647358Z 
2025-07-17T11:07:25.5647447Z SIGABRT(6), PID: 1777271, Thread 1777658: 
2025-07-17T11:07:25.5647917Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5648125Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5648336Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5648536Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5649095Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5649307Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5649583Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5649587Z 
2025-07-17T11:07:25.5649680Z SIGABRT(6), PID: 1777265, Thread 1777293: 
2025-07-17T11:07:25.5650155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5650429Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5650639Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5650837Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5651341Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5651558Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5651768Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5651772Z 
2025-07-17T11:07:25.5651864Z SIGABRT(6), PID: 1777271, Thread 1777663: 
2025-07-17T11:07:25.5652327Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5652536Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5652738Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5652937Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5653431Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5653711Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5653922Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5653927Z 
2025-07-17T11:07:25.5654024Z SIGABRT(6), PID: 1777265, Thread 1777294: 
2025-07-17T11:07:25.5654489Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5654702Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5654907Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5655112Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5655605Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5655818Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5656109Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5656115Z 
2025-07-17T11:07:25.5656208Z SIGABRT(6), PID: 1777271, Thread 1777667: 
2025-07-17T11:07:25.5656676Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5656886Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5657170Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5657375Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5657949Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5658164Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5658372Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5658450Z 
2025-07-17T11:07:25.5658546Z SIGABRT(6), PID: 1777265, Thread 1777295: 
2025-07-17T11:07:25.5659012Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5659222Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5659432Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5659636Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5660130Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5660341Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5660550Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5660554Z 
2025-07-17T11:07:25.5660647Z SIGABRT(6), PID: 1777271, Thread 1777670: 
2025-07-17T11:07:25.5661110Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5661322Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5661609Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5661812Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5662303Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5662518Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5662730Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5662735Z 
2025-07-17T11:07:25.5662827Z SIGABRT(6), PID: 1777271, Thread 1777674: 
2025-07-17T11:07:25.5663295Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5663500Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5663708Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5663905Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5664398Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5664605Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5664815Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5664819Z 
2025-07-17T11:07:25.5664909Z SIGABRT(6), PID: 1777265, Thread 1777296: 
2025-07-17T11:07:25.5665457Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5665734Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5665946Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5666145Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5666641Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5666911Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5667127Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5667130Z 
2025-07-17T11:07:25.5667218Z SIGABRT(6), PID: 1777271, Thread 1777677: 
2025-07-17T11:07:25.5667693Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5667901Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5668105Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5668298Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5668799Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5669004Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5669216Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5669220Z 
2025-07-17T11:07:25.5669311Z SIGABRT(6), PID: 1777265, Thread 1777298: 
2025-07-17T11:07:25.5669782Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5670058Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5670267Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5670464Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5670959Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5671166Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5671387Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5671392Z 
2025-07-17T11:07:25.5671490Z SIGABRT(6), PID: 1777267, Thread 1781621: 
2025-07-17T11:07:25.5671960Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5672177Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5672386Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5672596Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5672973Z frame #4: <unknown function> + 0x5a02023 (0x7f87ef061023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5673616Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87ef0651ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5674273Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87ef05c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5674667Z frame #7: <unknown function> + 0x79d655 (0x7f880054c655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5674903Z frame #8: <unknown function> + 0xdc253 (0x7f87b7597253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5675209Z frame #9: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5675431Z frame #10: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5675435Z 
2025-07-17T11:07:25.5675537Z SIGABRT(6), PID: 1777271, Thread 1777681: 
2025-07-17T11:07:25.5676010Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5676230Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5676432Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5676635Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5677131Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5677344Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5677553Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5677562Z 
2025-07-17T11:07:25.5677654Z SIGABRT(6), PID: 1777265, Thread 1777300: 
2025-07-17T11:07:25.5678131Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5678416Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5678629Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5678826Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5679331Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5679542Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5679760Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5679764Z 
2025-07-17T11:07:25.5679857Z SIGABRT(6), PID: 1777271, Thread 1777685: 
2025-07-17T11:07:25.5680333Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5680540Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5680752Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5680951Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5681448Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5681654Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5681939Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5681944Z 
2025-07-17T11:07:25.5682045Z SIGABRT(6), PID: 1777265, Thread 1777301: 
2025-07-17T11:07:25.5682581Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5682793Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5683008Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5683273Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5683775Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5683986Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5684210Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5684215Z 
2025-07-17T11:07:25.5684309Z SIGABRT(6), PID: 1777271, Thread 1777687: 
2025-07-17T11:07:25.5684784Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5684990Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5685197Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5685394Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5685897Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5686105Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5686390Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5686483Z SIGABRT(6), PID: 1777265, Thread 1777303: 
2025-07-17T11:07:25.5686956Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5687164Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5687374Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5687575Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5688073Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5688281Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5688501Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5688505Z 
2025-07-17T11:07:25.5688508Z 
2025-07-17T11:07:25.5688607Z SIGABRT(6), PID: 1777266, Thread 1777266: 
2025-07-17T11:07:25.5689072Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5689286Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5689481Z frame #2: pthread_kill + 0x12c (0x7fa32eeae9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5689658Z frame #3: raise + 0x16 (0x7fa32ee5a476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5690101Z frame #4: <unknown function> + 0x4feaa4d (0x7fa18bb33a4d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/_C/libtriton.so)
2025-07-17T11:07:25.5690381Z frame #5: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5690626Z frame #6: <unknown function> + 0x7741f (0x7fa1ab10941f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5690869Z frame #7: <unknown function> + 0x76c8e (0x7fa1ab108c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5691100Z frame #8: <unknown function> + 0x6ad69 (0x7fa1ab0fcd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5691390Z frame #9: <unknown function> + 0x3f54db (0x7fa23bd614db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5691614Z frame #10: <unknown function> + 0x3e11d8 (0x7fa23bd4d1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5691838Z frame #11: <unknown function> + 0xb2f86 (0x7fa23ba1ef86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5692046Z frame #12: <unknown function> + 0x8e501 (0x7fa23b9fa501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5692442Z frame #13: c10::hip::device_synchronize() + 0x5a (0x7fa2668017ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.5692832Z frame #14: <unknown function> + 0xd307d9 (0x7fa284d547d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5693065Z frame #15: <unknown function> + 0x139902 (0x55b7468fc902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5693318Z frame #16: _PyEval_EvalFrameDefault + 0x49b5 (0x55b7468f3a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5693558Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5693807Z frame #18: _PyEval_EvalFrameDefault + 0x49b5 (0x55b7468f3a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5694037Z frame #19: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5694283Z frame #20: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5694585Z frame #21: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5694825Z frame #22: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5695050Z frame #23: <unknown function> + 0x1474b2 (0x55b74690a4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5695256Z frame #24: PyObject_Call + 0xbc (0x55b74690ae4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5695486Z frame #25: <unknown function> + 0x1ff69d (0x55b7469c269d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5695713Z frame #26: _PyObject_MakeTpCall + 0x2d3 (0x55b7468f8023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5696012Z frame #27: _PyEval_EvalFrameDefault + 0x4c16 (0x55b7468f3cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5696241Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5696488Z frame #29: _PyEval_EvalFrameDefault + 0x30c (0x55b7468ef3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5696715Z frame #30: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5696955Z frame #31: _PyEval_EvalFrameDefault + 0x30c (0x55b7468ef3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5697175Z frame #32: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5697422Z frame #33: _PyEval_EvalFrameDefault + 0x49b5 (0x55b7468f3a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5697677Z frame #34: _PyObject_FastCallDictTstate + 0xd0 (0x55b7468f74b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5697900Z frame #35: _PyObject_Call_Prepend + 0x69 (0x55b746908ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5698206Z frame #36: <unknown function> + 0x205949 (0x55b7469c8949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5698665Z frame #37: _PyObject_MakeTpCall + 0x2d3 (0x55b7468f8023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5698913Z frame #38: _PyEval_EvalFrameDefault + 0x4c16 (0x55b7468f3cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5699145Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5699382Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55b7468ef3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5699687Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5699923Z frame #42: _PyEval_EvalFrameDefault + 0x30c (0x55b7468ef3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5700151Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5700389Z frame #44: _PyEval_EvalFrameDefault + 0x30c (0x55b7468ef3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5700614Z frame #45: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5700865Z frame #46: _PyEval_EvalFrameDefault + 0x1340 (0x55b7468f03f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5701096Z frame #47: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5701488Z frame #48: <unknown function> + 0x7ab3f7 (0x7fa2847cf3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5701714Z frame #49: <unknown function> + 0x13bc88 (0x55b7468fec88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5701923Z frame #50: PyObject_Call + 0x20f (0x55b74690af9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5702174Z frame #51: _PyEval_EvalFrameDefault + 0x58dd (0x55b7468f498d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5702400Z frame #52: <unknown function> + 0x147608 (0x55b74690a608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5702738Z frame #53: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b7468f1c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5702962Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5703206Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55b7468ef3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5703433Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5703681Z frame #57: _PyEval_EvalFrameDefault + 0x1340 (0x55b7468f03f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5703905Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5704147Z frame #59: _PyEval_EvalFrameDefault + 0x30c (0x55b7468ef3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5704371Z frame #60: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5704622Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b7468f1c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5704844Z frame #62: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5704849Z 
2025-07-17T11:07:25.5704949Z SIGABRT(6), PID: 1777271, Thread 1777690: 
2025-07-17T11:07:25.5705429Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5705653Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5705860Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5706069Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5706635Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5706937Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5707160Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5707260Z SIGABRT(6), PID: 1777265, Thread 1777305: 
2025-07-17T11:07:25.5707729Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5708009Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5708216Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5708425Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5708926Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5709139Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5709353Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5709358Z 
2025-07-17T11:07:25.5709361Z 
2025-07-17T11:07:25.5709459Z SIGABRT(6), PID: 1777272, Thread 1777272: 
2025-07-17T11:07:25.5709935Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5710152Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5710346Z frame #2: pthread_kill + 0x12c (0x7f1909ec99fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5710521Z frame #3: raise + 0x16 (0x7f1909e75476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5710988Z frame #4: <unknown function> + 0x4feaa4d (0x7f1766b51a4d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/_C/libtriton.so)
2025-07-17T11:07:25.5711203Z frame #5: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5711435Z frame #6: <unknown function> + 0x7741f (0x7f178612741f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5711671Z frame #7: <unknown function> + 0x76c8e (0x7f1786126c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5711897Z frame #8: <unknown function> + 0x6ad69 (0x7f178611ad69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.5712124Z frame #9: <unknown function> + 0x3f54db (0x7f1816d7f4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5712341Z frame #10: <unknown function> + 0x3e11d8 (0x7f1816d6b1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5712561Z frame #11: <unknown function> + 0xb2f86 (0x7f1816a3cf86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5712771Z frame #12: <unknown function> + 0x8e501 (0x7f1816a18501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.5713158Z frame #13: c10::hip::device_synchronize() + 0x5a (0x7f184181f7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.5713540Z frame #14: <unknown function> + 0xd307d9 (0x7f185fd727d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5713774Z frame #15: <unknown function> + 0x139902 (0x56197ceab902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5714026Z frame #16: _PyEval_EvalFrameDefault + 0x49b5 (0x56197cea2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5714263Z frame #17: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5714569Z frame #18: _PyEval_EvalFrameDefault + 0x49b5 (0x56197cea2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5714808Z frame #19: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5715108Z frame #20: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5715339Z frame #21: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5715576Z frame #22: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5715863Z frame #23: <unknown function> + 0x1474b2 (0x56197ceb94b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5716069Z frame #24: PyObject_Call + 0xbc (0x56197ceb9e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5716294Z frame #25: <unknown function> + 0x1ff69d (0x56197cf7169d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5716518Z frame #26: _PyObject_MakeTpCall + 0x2d3 (0x56197cea7023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5716771Z frame #27: _PyEval_EvalFrameDefault + 0x4c16 (0x56197cea2cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5717000Z frame #28: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5717243Z frame #29: _PyEval_EvalFrameDefault + 0x30c (0x56197ce9e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5717471Z frame #30: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5717713Z frame #31: _PyEval_EvalFrameDefault + 0x30c (0x56197ce9e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5717938Z frame #32: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5718184Z frame #33: _PyEval_EvalFrameDefault + 0x49b5 (0x56197cea2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5718442Z frame #34: _PyObject_FastCallDictTstate + 0xd0 (0x56197cea64b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5718675Z frame #35: _PyObject_Call_Prepend + 0x69 (0x56197ceb7ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5718967Z frame #36: <unknown function> + 0x205949 (0x56197cf77949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5719204Z frame #37: _PyObject_MakeTpCall + 0x2d3 (0x56197cea7023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5719445Z frame #38: _PyEval_EvalFrameDefault + 0x4c16 (0x56197cea2cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5719678Z frame #39: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5719913Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x56197ce9e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5720146Z frame #41: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5720382Z frame #42: _PyEval_EvalFrameDefault + 0x30c (0x56197ce9e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5720616Z frame #43: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5720854Z frame #44: _PyEval_EvalFrameDefault + 0x30c (0x56197ce9e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5721085Z frame #45: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5721325Z frame #46: _PyEval_EvalFrameDefault + 0x1340 (0x56197ce9f3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5721553Z frame #47: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5721938Z frame #48: <unknown function> + 0x7ab3f7 (0x7f185f7ed3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5722167Z frame #49: <unknown function> + 0x13bc88 (0x56197ceadc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5722372Z frame #50: PyObject_Call + 0x20f (0x56197ceb9f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5722687Z frame #51: _PyEval_EvalFrameDefault + 0x58dd (0x56197cea398d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5722972Z frame #52: <unknown function> + 0x147608 (0x56197ceb9608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5723223Z frame #53: _PyEval_EvalFrameDefault + 0x2ba5 (0x56197cea0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5723452Z frame #54: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5723762Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x56197ce9e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5723987Z frame #56: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5724231Z frame #57: _PyEval_EvalFrameDefault + 0x1340 (0x56197ce9f3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5724456Z frame #58: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5724699Z frame #59: _PyEval_EvalFrameDefault + 0x30c (0x56197ce9e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5724926Z frame #60: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5725173Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x56197cea0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5725398Z frame #62: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.5725403Z 
2025-07-17T11:07:25.5725502Z SIGABRT(6), PID: 1777266, Thread 1777297: 
2025-07-17T11:07:25.5725983Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5726206Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5726418Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5726632Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5727207Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5727430Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5727644Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5727649Z 
2025-07-17T11:07:25.5727748Z SIGABRT(6), PID: 1777271, Thread 1777692: 
2025-07-17T11:07:25.5728216Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5728427Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5728632Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5728840Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5729338Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5729555Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5729766Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5729776Z 
2025-07-17T11:07:25.5729865Z SIGABRT(6), PID: 1777265, Thread 1777307: 
2025-07-17T11:07:25.5730337Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5730610Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5730827Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5731086Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5731592Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5731800Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5732082Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5732087Z 
2025-07-17T11:07:25.5732179Z SIGABRT(6), PID: 1777272, Thread 1777627: 
2025-07-17T11:07:25.5732652Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5732860Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5733073Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5733269Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5733768Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5733976Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5734195Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5734199Z 
2025-07-17T11:07:25.5734289Z SIGABRT(6), PID: 1777266, Thread 1777299: 
2025-07-17T11:07:25.5734763Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5735045Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5735258Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5735463Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5736052Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5736267Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5736491Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5736495Z 
2025-07-17T11:07:25.5736587Z SIGABRT(6), PID: 1777271, Thread 1777695: 
2025-07-17T11:07:25.5737065Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5737274Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5737484Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5737679Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5738182Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5738388Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5738599Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5738603Z 
2025-07-17T11:07:25.5738806Z SIGABRT(6), PID: 1777265, Thread 1777309: 
2025-07-17T11:07:25.5739385Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5739614Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5739832Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5740039Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5740633Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5740847Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5741065Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5741071Z 
2025-07-17T11:07:25.5741164Z SIGABRT(6), PID: 1777267, Thread 1781622: 
2025-07-17T11:07:25.5741649Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5741866Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5742072Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5742276Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5742655Z frame #4: <unknown function> + 0x5a02023 (0x7f87ef061023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5743238Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87ef0651ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5743834Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87ef05c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5744307Z frame #7: <unknown function> + 0x79d655 (0x7f880054c655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5744540Z frame #8: <unknown function> + 0xdc253 (0x7f87b7597253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5744760Z frame #9: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5744979Z frame #10: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5744984Z 
2025-07-17T11:07:25.5745085Z SIGABRT(6), PID: 1777272, Thread 1777632: 
2025-07-17T11:07:25.5745560Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5745780Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5745986Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5746194Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5746692Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5746910Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5747121Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5747125Z 
2025-07-17T11:07:25.5747224Z SIGABRT(6), PID: 1777271, Thread 1777698: 
2025-07-17T11:07:25.5747813Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5748040Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5748245Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5748451Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5749023Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5749241Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5749451Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5749455Z 
2025-07-17T11:07:25.5749553Z SIGABRT(6), PID: 1777266, Thread 1777302: 
2025-07-17T11:07:25.5750028Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5750246Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5750452Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5750660Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5751158Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5751375Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5751590Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5751599Z 
2025-07-17T11:07:25.5751759Z SIGABRT(6), PID: 1777265, Thread 1777311: 
2025-07-17T11:07:25.5752237Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5752448Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5752659Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5752860Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5753362Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5753571Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5753787Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5753881Z SIGABRT(6), PID: 1777272, Thread 1777636: 
2025-07-17T11:07:25.5754355Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5754562Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5754768Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5754964Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5755460Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5755732Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5755954Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5756018Z 
2025-07-17T11:07:25.5756022Z 
2025-07-17T11:07:25.5756119Z SIGABRT(6), PID: 1777271, Thread 1777700: 
2025-07-17T11:07:25.5756593Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5756796Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5757072Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5757272Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5757777Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5757985Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5758204Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5758294Z SIGABRT(6), PID: 1777266, Thread 1777304: 
2025-07-17T11:07:25.5758772Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5758983Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5759194Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5759392Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5759894Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5760177Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5760394Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5760400Z 
2025-07-17T11:07:25.5760403Z 
2025-07-17T11:07:25.5760493Z SIGABRT(6), PID: 1777272, Thread 1777639: 
2025-07-17T11:07:25.5760961Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5761167Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5761376Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5761573Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5762076Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5762295Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5762508Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5762511Z 
2025-07-17T11:07:25.5762603Z SIGABRT(6), PID: 1777265, Thread 1777313: 
2025-07-17T11:07:25.5763072Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5763289Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5763495Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5763768Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5764326Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5764544Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5764753Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5764757Z 
2025-07-17T11:07:25.5764916Z SIGABRT(6), PID: 1777266, Thread 1777306: 
2025-07-17T11:07:25.5765381Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5765597Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5765802Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5766009Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5766505Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5766715Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5766925Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5766930Z 
2025-07-17T11:07:25.5767027Z SIGABRT(6), PID: 1777271, Thread 1777702: 
2025-07-17T11:07:25.5767490Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5767703Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5767908Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5768203Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5768694Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5768905Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5769117Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5769122Z 
2025-07-17T11:07:25.5769220Z SIGABRT(6), PID: 1777272, Thread 1777643: 
2025-07-17T11:07:25.5769685Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5769900Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5770105Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5770312Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5770802Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5771015Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5771228Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5771232Z 
2025-07-17T11:07:25.5771327Z SIGABRT(6), PID: 1777265, Thread 1777316: 
2025-07-17T11:07:25.5771790Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5772075Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5772353Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5772562Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5773061Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5773333Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5773550Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5773554Z 
2025-07-17T11:07:25.5773647Z SIGABRT(6), PID: 1777271, Thread 1777705: 
2025-07-17T11:07:25.5774121Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5774332Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5774544Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5774742Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5775236Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5775443Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5775658Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5775748Z SIGABRT(6), PID: 1777266, Thread 1777308: 
2025-07-17T11:07:25.5776283Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5776584Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5776800Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5776998Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5777505Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5777715Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5777935Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5777941Z 
2025-07-17T11:07:25.5777944Z 
2025-07-17T11:07:25.5778038Z SIGABRT(6), PID: 1777272, Thread 1777647: 
2025-07-17T11:07:25.5778510Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5778717Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5778927Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5779126Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5779632Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5779839Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5780053Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5780136Z 
2025-07-17T11:07:25.5780230Z SIGABRT(6), PID: 1777265, Thread 1777317: 
2025-07-17T11:07:25.5780780Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5780993Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5781202Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5781477Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5781980Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5782190Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5782409Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5782415Z 
2025-07-17T11:07:25.5782508Z SIGABRT(6), PID: 1777271, Thread 1777707: 
2025-07-17T11:07:25.5782982Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5783197Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5783400Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5783604Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5784096Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5784313Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5784523Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5784596Z 
2025-07-17T11:07:25.5784697Z SIGABRT(6), PID: 1777266, Thread 1777310: 
2025-07-17T11:07:25.5785169Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5785385Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5785595Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5785798Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5786291Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5786511Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5786730Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5786734Z 
2025-07-17T11:07:25.5786831Z SIGABRT(6), PID: 1777272, Thread 1777650: 
2025-07-17T11:07:25.5787294Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5787507Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5787711Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5787915Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5788478Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5788766Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5788978Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5789078Z SIGABRT(6), PID: 1777265, Thread 1777319: 
2025-07-17T11:07:25.5789542Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5789846Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5790054Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5790260Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5790753Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5790973Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5791188Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5791192Z 
2025-07-17T11:07:25.5791195Z 
2025-07-17T11:07:25.5791293Z SIGABRT(6), PID: 1777271, Thread 1777709: 
2025-07-17T11:07:25.5791757Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5791968Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5792172Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5792378Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5792875Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5793158Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5793369Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5793380Z 
2025-07-17T11:07:25.5793470Z SIGABRT(6), PID: 1777265, Thread 1777321: 
2025-07-17T11:07:25.5793946Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5794156Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5794369Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5794572Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5795076Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5795284Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5795501Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5795507Z 
2025-07-17T11:07:25.5795597Z SIGABRT(6), PID: 1777267, Thread 1781623: 
2025-07-17T11:07:25.5796069Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5796278Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5796559Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5796823Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5797205Z frame #4: <unknown function> + 0x5a02023 (0x7f87ef061023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5797774Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87ef0651ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5798459Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87ef05c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5798840Z frame #7: <unknown function> + 0x79d655 (0x7f880054c655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5799080Z frame #8: <unknown function> + 0xdc253 (0x7f87b7597253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5799294Z frame #9: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5799520Z frame #10: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5799616Z SIGABRT(6), PID: 1777272, Thread 1777654: 
2025-07-17T11:07:25.5800091Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5800302Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5800514Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5800714Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5801220Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5801503Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5801722Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5801814Z SIGABRT(6), PID: 1777266, Thread 1777312: 
2025-07-17T11:07:25.5802288Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5802501Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5802716Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5802917Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5803425Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5803641Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5803857Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5803861Z 
2025-07-17T11:07:25.5803864Z 
2025-07-17T11:07:25.5803867Z 
2025-07-17T11:07:25.5803960Z SIGABRT(6), PID: 1777271, Thread 1777711: 
2025-07-17T11:07:25.5804437Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5804649Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5804855Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5805123Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5805682Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5805900Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5806117Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5806181Z 
2025-07-17T11:07:25.5806282Z SIGABRT(6), PID: 1777265, Thread 1777323: 
2025-07-17T11:07:25.5806750Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5806964Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5807171Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5807384Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5807882Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5808097Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5808310Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5808314Z 
2025-07-17T11:07:25.5808409Z SIGABRT(6), PID: 1777272, Thread 1777659: 
2025-07-17T11:07:25.5808879Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5809097Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5809306Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5809582Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5810080Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5810298Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5810507Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5810511Z 
2025-07-17T11:07:25.5810607Z SIGABRT(6), PID: 1777266, Thread 1777314: 
2025-07-17T11:07:25.5811068Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5811282Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5811492Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5811698Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5812187Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5812405Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5812615Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5812619Z 
2025-07-17T11:07:25.5812718Z SIGABRT(6), PID: 1777271, Thread 1777713: 
2025-07-17T11:07:25.5813252Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5813530Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5813740Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5813941Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5814436Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5814712Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5814921Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5814935Z 
2025-07-17T11:07:25.5815027Z SIGABRT(6), PID: 1777265, Thread 1777325: 
2025-07-17T11:07:25.5815499Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5815713Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5815975Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5816176Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5816674Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5816885Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5817101Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5817105Z 
2025-07-17T11:07:25.5817196Z SIGABRT(6), PID: 1777272, Thread 1777662: 
2025-07-17T11:07:25.5817672Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5817965Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5818179Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5818376Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5818874Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5819079Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5819293Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5819385Z SIGABRT(6), PID: 1777271, Thread 1777714: 
2025-07-17T11:07:25.5819863Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5820068Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5820274Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5820472Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5820966Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5821174Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5821462Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5821470Z 
2025-07-17T11:07:25.5821474Z 
2025-07-17T11:07:25.5821641Z SIGABRT(6), PID: 1777266, Thread 1777315: 
2025-07-17T11:07:25.5822119Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5822327Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5822618Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5822820Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5823321Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5823531Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5823750Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5823758Z 
2025-07-17T11:07:25.5823850Z SIGABRT(6), PID: 1777265, Thread 1777327: 
2025-07-17T11:07:25.5824329Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5824536Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5824744Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5824944Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5825443Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5825655Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5825945Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5825949Z 
2025-07-17T11:07:25.5826046Z SIGABRT(6), PID: 1777272, Thread 1777666: 
2025-07-17T11:07:25.5826510Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5826725Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5826928Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5827130Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5827624Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5827844Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5828048Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5828052Z 
2025-07-17T11:07:25.5828152Z SIGABRT(6), PID: 1777271, Thread 1777716: 
2025-07-17T11:07:25.5828617Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5828835Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5829038Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5829244Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5829946Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5830246Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5830460Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5830559Z SIGABRT(6), PID: 1777266, Thread 1777318: 
2025-07-17T11:07:25.5831025Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5831306Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5831519Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5831724Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5832220Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5832441Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5832652Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5832656Z 
2025-07-17T11:07:25.5832659Z 
2025-07-17T11:07:25.5832760Z SIGABRT(6), PID: 1777265, Thread 1777329: 
2025-07-17T11:07:25.5833223Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5833438Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5833645Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5833848Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5834414Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5834633Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5834843Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5834849Z 
2025-07-17T11:07:25.5834949Z SIGABRT(6), PID: 1777272, Thread 1777669: 
2025-07-17T11:07:25.5835413Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5835628Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5835834Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5836038Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5836533Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5836748Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5836961Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5836971Z 
2025-07-17T11:07:25.5837061Z SIGABRT(6), PID: 1777266, Thread 1777320: 
2025-07-17T11:07:25.5837530Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5837806Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5838022Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5838285Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5838784Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5838996Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5839275Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5839369Z SIGABRT(6), PID: 1777271, Thread 1777717: 
2025-07-17T11:07:25.5839841Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5840047Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5840257Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5840452Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5840947Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5841158Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5841374Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5841378Z 
2025-07-17T11:07:25.5841381Z 
2025-07-17T11:07:25.5841475Z SIGABRT(6), PID: 1777265, Thread 1777331: 
2025-07-17T11:07:25.5841953Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5842237Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5842454Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5842658Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5843171Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5843387Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5843608Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5843612Z 
2025-07-17T11:07:25.5843702Z SIGABRT(6), PID: 1777272, Thread 1777673: 
2025-07-17T11:07:25.5844179Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5844390Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5844599Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5844795Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5845294Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5845505Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5845718Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5845722Z 
2025-07-17T11:07:25.5845881Z SIGABRT(6), PID: 1777271, Thread 1777719: 
2025-07-17T11:07:25.5846417Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5846629Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5846838Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5847039Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5847601Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5847814Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5848023Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5848029Z 
2025-07-17T11:07:25.5848131Z SIGABRT(6), PID: 1777267, Thread 1781624: 
2025-07-17T11:07:25.5848603Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5848819Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5849025Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5849236Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5849613Z frame #4: <unknown function> + 0x5a02023 (0x7f87ef061023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5850188Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87ef0651ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5850781Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87ef05c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5851237Z frame #7: <unknown function> + 0x79d655 (0x7f880054c655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5851471Z frame #8: <unknown function> + 0xdc253 (0x7f87b7597253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5851689Z frame #9: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5851911Z frame #10: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5851915Z 
2025-07-17T11:07:25.5852011Z SIGABRT(6), PID: 1777265, Thread 1777333: 
2025-07-17T11:07:25.5852487Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5852705Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5852913Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5853122Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5853623Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5853841Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5854051Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5854055Z 
2025-07-17T11:07:25.5854150Z SIGABRT(6), PID: 1777266, Thread 1777322: 
2025-07-17T11:07:25.5854679Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5854959Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5855168Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5855372Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5855976Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5856194Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5856406Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5856411Z 
2025-07-17T11:07:25.5856509Z SIGABRT(6), PID: 1777271, Thread 1777721: 
2025-07-17T11:07:25.5856983Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5857201Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5857416Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5857616Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5858120Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5858329Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5858543Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5858548Z 
2025-07-17T11:07:25.5858640Z SIGABRT(6), PID: 1777272, Thread 1777676: 
2025-07-17T11:07:25.5859199Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5859403Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5859615Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5859814Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5860311Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5860519Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5860736Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5860741Z 
2025-07-17T11:07:25.5860835Z SIGABRT(6), PID: 1777265, Thread 1777335: 
2025-07-17T11:07:25.5861308Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5861514Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5861728Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5861924Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5862420Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5862729Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5862954Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5863033Z 
2025-07-17T11:07:25.5863132Z SIGABRT(6), PID: 1777266, Thread 1777324: 
2025-07-17T11:07:25.5863609Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5863818Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5864112Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5864313Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5864815Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5865028Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5865248Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5865252Z 
2025-07-17T11:07:25.5865343Z SIGABRT(6), PID: 1777271, Thread 1777722: 
2025-07-17T11:07:25.5865821Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5866027Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5866243Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5866445Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5866958Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5867239Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5867455Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5867461Z 
2025-07-17T11:07:25.5867554Z SIGABRT(6), PID: 1777272, Thread 1777680: 
2025-07-17T11:07:25.5868030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5868248Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5868455Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5868660Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5869170Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5869394Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5869605Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5869609Z 
2025-07-17T11:07:25.5869707Z SIGABRT(6), PID: 1777265, Thread 1777336: 
2025-07-17T11:07:25.5870178Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5870396Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5870602Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5870879Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5871459Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5871685Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5871900Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5871904Z 
2025-07-17T11:07:25.5872067Z SIGABRT(6), PID: 1777266, Thread 1777326: 
2025-07-17T11:07:25.5872538Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5872755Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5872964Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5873171Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5873671Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5873889Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5874101Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5874107Z 
2025-07-17T11:07:25.5874200Z SIGABRT(6), PID: 1777271, Thread 1777724: 
2025-07-17T11:07:25.5874666Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5874881Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5875087Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5875367Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5875858Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5876074Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5876283Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5876287Z 
2025-07-17T11:07:25.5876383Z SIGABRT(6), PID: 1777272, Thread 1777684: 
2025-07-17T11:07:25.5876859Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5877075Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5877281Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5877487Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5877982Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5878200Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5878409Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5878419Z 
2025-07-17T11:07:25.5878511Z SIGABRT(6), PID: 1777265, Thread 1777338: 
2025-07-17T11:07:25.5878988Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5879269Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5879547Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5879750Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5880252Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5880523Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5880744Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5880748Z 
2025-07-17T11:07:25.5880838Z SIGABRT(6), PID: 1777271, Thread 1777726: 
2025-07-17T11:07:25.5881315Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5881525Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5881737Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5881941Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5882441Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5882651Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5882867Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5882871Z 
2025-07-17T11:07:25.5882963Z SIGABRT(6), PID: 1777266, Thread 1777328: 
2025-07-17T11:07:25.5883443Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5883726Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5883943Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5884140Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5884640Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5884852Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5885066Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5885070Z 
2025-07-17T11:07:25.5885163Z SIGABRT(6), PID: 1777272, Thread 1777688: 
2025-07-17T11:07:25.5885639Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5885847Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5886055Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5886250Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5886750Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5886959Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5887188Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5887259Z 
2025-07-17T11:07:25.5887355Z SIGABRT(6), PID: 1777265, Thread 1777340: 
2025-07-17T11:07:25.5887894Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5888104Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5888318Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5888583Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5889086Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5889302Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5889517Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5889523Z 
2025-07-17T11:07:25.5889617Z SIGABRT(6), PID: 1777266, Thread 1777330: 
2025-07-17T11:07:25.5890087Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5890300Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5890505Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5890715Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5891215Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5891431Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5891643Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5891737Z 
2025-07-17T11:07:25.5891843Z SIGABRT(6), PID: 1777271, Thread 1777728: 
2025-07-17T11:07:25.5892308Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5892521Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5892726Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5892927Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5893417Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5893629Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5893840Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5893937Z SIGABRT(6), PID: 1777272, Thread 1777691: 
2025-07-17T11:07:25.5894403Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5894615Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5894821Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5895025Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5895583Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5895803Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5896159Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5896165Z 
2025-07-17T11:07:25.5896168Z 
2025-07-17T11:07:25.5896275Z SIGABRT(6), PID: 1777265, Thread 1777342: 
2025-07-17T11:07:25.5896740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5897032Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5897239Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5897445Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5897941Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5898161Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5898369Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5898373Z 
2025-07-17T11:07:25.5898471Z SIGABRT(6), PID: 1777272, Thread 1777693: 
2025-07-17T11:07:25.5898941Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5899155Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5899362Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5899564Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5900056Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5900357Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5900566Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5900577Z 
2025-07-17T11:07:25.5900672Z SIGABRT(6), PID: 1777271, Thread 1777729: 
2025-07-17T11:07:25.5901147Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5901353Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5901563Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5901760Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5902265Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5902473Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5902691Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5902696Z 
2025-07-17T11:07:25.5902789Z SIGABRT(6), PID: 1777266, Thread 1777332: 
2025-07-17T11:07:25.5903264Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5903472Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5903766Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5903966Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5904530Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5904743Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5904962Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5905025Z 
2025-07-17T11:07:25.5905118Z SIGABRT(6), PID: 1777265, Thread 1777344: 
2025-07-17T11:07:25.5905594Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5905802Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5906019Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5906224Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5906728Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5906943Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5907161Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5907165Z 
2025-07-17T11:07:25.5907256Z SIGABRT(6), PID: 1777267, Thread 1781625: 
2025-07-17T11:07:25.5907731Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5907940Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5908225Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5908425Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5908813Z frame #4: <unknown function> + 0x5a02023 (0x7f87ef061023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5909377Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87ef0651ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5909973Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87ef05c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5910358Z frame #7: <unknown function> + 0x79d655 (0x7f880054c655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5910605Z frame #8: <unknown function> + 0xdc253 (0x7f87b7597253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5910814Z frame #9: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5911036Z frame #10: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5911040Z 
2025-07-17T11:07:25.5911140Z SIGABRT(6), PID: 1777272, Thread 1777696: 
2025-07-17T11:07:25.5911615Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5911827Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5912031Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5912300Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5912867Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5913087Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5913297Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5913359Z 
2025-07-17T11:07:25.5913461Z SIGABRT(6), PID: 1777271, Thread 1777730: 
2025-07-17T11:07:25.5913925Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5914137Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5914342Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5914551Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5915050Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5915267Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5915477Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5915572Z SIGABRT(6), PID: 1777266, Thread 1777334: 
2025-07-17T11:07:25.5916040Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5916255Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5916462Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5916738Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5917234Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5917457Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5917673Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5917677Z 
2025-07-17T11:07:25.5917681Z 
2025-07-17T11:07:25.5917781Z SIGABRT(6), PID: 1777265, Thread 1777346: 
2025-07-17T11:07:25.5918249Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5918467Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5918674Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5918886Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5919382Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5919601Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5919809Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5919813Z 
2025-07-17T11:07:25.5919907Z SIGABRT(6), PID: 1777272, Thread 1777701: 
2025-07-17T11:07:25.5920437Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5920715Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5920927Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5921130Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5921627Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5921927Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5922143Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5922147Z 
2025-07-17T11:07:25.5922240Z SIGABRT(6), PID: 1777271, Thread 1777731: 
2025-07-17T11:07:25.5922714Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5922923Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5923137Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5923332Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5923833Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5924043Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5924261Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5924356Z SIGABRT(6), PID: 1777265, Thread 1777348: 
2025-07-17T11:07:25.5924835Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5925109Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5925326Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5925524Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5926029Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5926240Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5926453Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5926457Z 
2025-07-17T11:07:25.5926462Z 
2025-07-17T11:07:25.5948425Z SIGABRT(6), PID: 1777266, Thread 1777337: 
2025-07-17T11:07:25.5949024Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5949274Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5949496Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5949721Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5950251Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5950469Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5950826Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5950837Z 
2025-07-17T11:07:25.5951026Z SIGABRT(6), PID: 1777272, Thread 1777704: 
2025-07-17T11:07:25.5951526Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5951751Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5951956Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5952259Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5952775Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5952988Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5953198Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5953207Z 
2025-07-17T11:07:25.5953306Z SIGABRT(6), PID: 1777271, Thread 1777732: 
2025-07-17T11:07:25.5953781Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5953991Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5954193Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5954396Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5954906Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5955121Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5955423Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5955428Z 
2025-07-17T11:07:25.5955525Z SIGABRT(6), PID: 1777265, Thread 1777350: 
2025-07-17T11:07:25.5956003Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5956215Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5956415Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5956618Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5957118Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5957329Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5957535Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5957541Z 
2025-07-17T11:07:25.5957631Z SIGABRT(6), PID: 1777266, Thread 1777339: 
2025-07-17T11:07:25.5958109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5958318Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5958525Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5958727Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5959330Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5959614Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5959832Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5959837Z 
2025-07-17T11:07:25.5959928Z SIGABRT(6), PID: 1777272, Thread 1777706: 
2025-07-17T11:07:25.5960410Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5960682Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5960885Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5961084Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5961593Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5961801Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5962013Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5962017Z 
2025-07-17T11:07:25.5962109Z SIGABRT(6), PID: 1777271, Thread 1777733: 
2025-07-17T11:07:25.5962584Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5962789Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5962995Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5963193Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5963764Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5963970Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5964175Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5964270Z SIGABRT(6), PID: 1777265, Thread 1777353: 
2025-07-17T11:07:25.5964738Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5964941Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5965146Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5965343Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5965843Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5966049Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5966260Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5966266Z 
2025-07-17T11:07:25.5966269Z 
2025-07-17T11:07:25.5966358Z SIGABRT(6), PID: 1777272, Thread 1777710: 
2025-07-17T11:07:25.5966822Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5967025Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5967310Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5967569Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5968068Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5968278Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5968550Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5968554Z 
2025-07-17T11:07:25.5968645Z SIGABRT(6), PID: 1777266, Thread 1777341: 
2025-07-17T11:07:25.5969111Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5969321Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5969526Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5969724Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5970221Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5970435Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5970644Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5970649Z 
2025-07-17T11:07:25.5970739Z SIGABRT(6), PID: 1777265, Thread 1777355: 
2025-07-17T11:07:25.5971201Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5971410Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5971690Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5971893Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5972389Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5972602Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5972810Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5972814Z 
2025-07-17T11:07:25.5972903Z SIGABRT(6), PID: 1777271, Thread 1777735: 
2025-07-17T11:07:25.5973362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5973574Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5973775Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5973973Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5974460Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5974668Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5974875Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5974880Z 
2025-07-17T11:07:25.5974969Z SIGABRT(6), PID: 1777272, Thread 1777712: 
2025-07-17T11:07:25.5975599Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5975813Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5976083Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5976284Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5976860Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5977071Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5977275Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5977279Z 
2025-07-17T11:07:25.5977373Z SIGABRT(6), PID: 1777266, Thread 1777343: 
2025-07-17T11:07:25.5977840Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5978050Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5978251Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5978455Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5978951Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5979160Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5979376Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5979381Z 
2025-07-17T11:07:25.5979552Z SIGABRT(6), PID: 1777271, Thread 1777737: 
2025-07-17T11:07:25.5980019Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5980227Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5980434Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5980632Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5981135Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5981339Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5981549Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5981554Z 
2025-07-17T11:07:25.5981643Z SIGABRT(6), PID: 1777265, Thread 1777357: 
2025-07-17T11:07:25.5982109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5982314Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5982521Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5982721Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5983218Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5983507Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5983797Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5983801Z 
2025-07-17T11:07:25.5983892Z SIGABRT(6), PID: 1777267, Thread 1781626: 
2025-07-17T11:07:25.5984361Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87b932ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5984565Z frame #1: <unknown function> + 0x42520 (0x7f88aabe0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5984836Z frame #2: <unknown function> + 0x91117 (0x7f88aac2f117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5985033Z frame #3: pthread_cond_wait + 0x211 (0x7f88aac31a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5985411Z frame #4: <unknown function> + 0x5a02023 (0x7f87ef061023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5985977Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87ef0651ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5986578Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87ef05c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.5986957Z frame #7: <unknown function> + 0x79d655 (0x7f880054c655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.5987194Z frame #8: <unknown function> + 0xdc253 (0x7f87b7597253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.5987403Z frame #9: <unknown function> + 0x94ac3 (0x7f88aac32ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5987627Z frame #10: <unknown function> + 0x126850 (0x7f88aacc4850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5987633Z 
2025-07-17T11:07:25.5987722Z SIGABRT(6), PID: 1777271, Thread 1777738: 
2025-07-17T11:07:25.5988266Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5988472Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5988672Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5988878Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5989378Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5989587Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5989794Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5989886Z SIGABRT(6), PID: 1777272, Thread 1777715: 
2025-07-17T11:07:25.5990351Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5990553Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5990752Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5990952Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5991442Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5991648Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5992046Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5992053Z 
2025-07-17T11:07:25.5992112Z 
2025-07-17T11:07:25.5992211Z SIGABRT(6), PID: 1777265, Thread 1777359: 
2025-07-17T11:07:25.5992679Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5992887Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5993151Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5993354Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5993847Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5994061Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5994276Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5994369Z SIGABRT(6), PID: 1777266, Thread 1777345: 
2025-07-17T11:07:25.5994831Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5995040Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5995244Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5995441Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5995935Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5996145Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5996424Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5996428Z 
2025-07-17T11:07:25.5996432Z 
2025-07-17T11:07:25.5996526Z SIGABRT(6), PID: 1777272, Thread 1777718: 
2025-07-17T11:07:25.5996986Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5997194Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5997395Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5997597Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5998090Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.5998299Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5998505Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5998516Z 
2025-07-17T11:07:25.5998605Z SIGABRT(6), PID: 1777271, Thread 1777741: 
2025-07-17T11:07:25.5999070Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.5999275Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5999477Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.5999671Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6000291Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6000502Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6000714Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6000718Z 
2025-07-17T11:07:25.6000806Z SIGABRT(6), PID: 1777265, Thread 1777360: 
2025-07-17T11:07:25.6001335Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6001540Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6001748Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6001946Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6002444Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6002648Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6002861Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6002866Z 
2025-07-17T11:07:25.6002955Z SIGABRT(6), PID: 1777266, Thread 1777347: 
2025-07-17T11:07:25.6003419Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6003624Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6003831Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6004029Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6004617Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6004827Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6005037Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6005128Z SIGABRT(6), PID: 1777272, Thread 1777720: 
2025-07-17T11:07:25.6005587Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6005790Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6005998Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6006194Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6006686Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6006890Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6007098Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6007184Z SIGABRT(6), PID: 1777271, Thread 1777743: 
2025-07-17T11:07:25.6007645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6007916Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6008126Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6008382Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6008879Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6009084Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6009358Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6009448Z SIGABRT(6), PID: 1777265, Thread 1777362: 
2025-07-17T11:07:25.6009915Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6010120Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6010329Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6010526Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6011020Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6011227Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6011438Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6011442Z 
2025-07-17T11:07:25.6011445Z 
2025-07-17T11:07:25.6011448Z 
2025-07-17T11:07:25.6011451Z 
2025-07-17T11:07:25.6011541Z SIGABRT(6), PID: 1777272, Thread 1777723: 
2025-07-17T11:07:25.6012006Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6012284Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6012488Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6012686Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6013174Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6013383Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6013591Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6013595Z 
2025-07-17T11:07:25.6013687Z SIGABRT(6), PID: 1777266, Thread 1777349: 
2025-07-17T11:07:25.6014155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6014363Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6014563Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6014761Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6015253Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6015465Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6015672Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6015677Z 
2025-07-17T11:07:25.6015833Z SIGABRT(6), PID: 1777265, Thread 1777364: 
2025-07-17T11:07:25.6016433Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6016647Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6016850Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6017126Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6017617Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6017827Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6018041Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6018046Z 
2025-07-17T11:07:25.6018153Z SIGABRT(6), PID: 1777271, Thread 1777744: 
2025-07-17T11:07:25.6018623Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6018836Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6019049Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6019256Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6019746Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6019962Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6020170Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6020260Z 
2025-07-17T11:07:25.6020355Z SIGABRT(6), PID: 1777272, Thread 1777725: 
2025-07-17T11:07:25.6020831Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6021036Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6021249Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6021448Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6021947Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6022156Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6022380Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6022385Z 
2025-07-17T11:07:25.6022476Z SIGABRT(6), PID: 1777266, Thread 1777351: 
2025-07-17T11:07:25.6022953Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6023157Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6023371Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6023572Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6024155Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6024374Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6024654Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6024659Z 
2025-07-17T11:07:25.6024750Z SIGABRT(6), PID: 1777265, Thread 1777367: 
2025-07-17T11:07:25.6025234Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6025526Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6025747Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6025947Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6026452Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6026665Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6026886Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6026978Z SIGABRT(6), PID: 1777271, Thread 1777746: 
2025-07-17T11:07:25.6027450Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6027659Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6027867Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6028065Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6028562Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6028842Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6029071Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6029075Z 
2025-07-17T11:07:25.6029078Z 
2025-07-17T11:07:25.6029170Z SIGABRT(6), PID: 1777272, Thread 1777727: 
2025-07-17T11:07:25.6029644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6029853Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6030068Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6030267Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6030769Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6030976Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6031192Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6031198Z 
2025-07-17T11:07:25.6031296Z SIGABRT(6), PID: 1777271, Thread 1777748: 
2025-07-17T11:07:25.6031761Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6031970Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6032238Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6032450Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6033008Z frame #4: <unknown function> + 0x3475ab (0x7f832d8185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6033231Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6033440Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6033529Z 
2025-07-17T11:07:25.6033630Z SIGABRT(6), PID: 1777265, Thread 1777369: 
2025-07-17T11:07:25.6034096Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6034313Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6034520Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6034735Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6035233Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6035447Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6035660Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6035758Z SIGABRT(6), PID: 1777266, Thread 1777352: 
2025-07-17T11:07:25.6036223Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6036441Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6036645Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6036926Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6037421Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6037635Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6037851Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6037854Z 
2025-07-17T11:07:25.6037858Z 
2025-07-17T11:07:25.6037956Z SIGABRT(6), PID: 1777272, Thread 1777734: 
2025-07-17T11:07:25.6038427Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6038643Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6038851Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6039061Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6039557Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6039779Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6039989Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6039993Z 
2025-07-17T11:07:25.6040094Z SIGABRT(6), PID: 1777271, Thread 1777787: 
2025-07-17T11:07:25.6040629Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6041155Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f823f07708a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6041376Z frame #2: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6041571Z frame #3: pthread_kill + 0x12c (0x7f83309829fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6041803Z frame #4: raise + 0x16 (0x7f833092e476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6041971Z frame #5: abort + 0xd3 (0x7f83309147f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6042211Z frame #6: <unknown function> + 0x972cc (0x7f81acc002cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6042449Z frame #7: <unknown function> + 0x95f47 (0x7f81acbfef47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6042675Z frame #8: <unknown function> + 0x34b71 (0x7f81acb9db71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6042888Z frame #9: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6043099Z frame #10: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6043107Z 
2025-07-17T11:07:25.6043200Z SIGABRT(6), PID: 1777266, Thread 1777354: 
2025-07-17T11:07:25.6043676Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6043886Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6044090Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6044293Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6044797Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6045077Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6045294Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6045298Z 
2025-07-17T11:07:25.6045387Z SIGABRT(6), PID: 1777265, Thread 1777371: 
2025-07-17T11:07:25.6045856Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6046064Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6046268Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6046467Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6046970Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6047181Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6047398Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6047402Z 
2025-07-17T11:07:25.6047490Z SIGABRT(6), PID: 1777272, Thread 1777736: 
2025-07-17T11:07:25.6047961Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6048165Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6048372Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6048634Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6049194Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6049401Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6049616Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6049681Z 
2025-07-17T11:07:25.6049772Z SIGABRT(6), PID: 1777271, Thread 1781113: 
2025-07-17T11:07:25.6050239Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6050444Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6050617Z frame #2: __poll + 0x4f (0x7f8330a04bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6051021Z frame #3: <unknown function> + 0x489f (0x7f833051089f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.6051260Z frame #4: <unknown function> + 0x1483bc (0x5616642273bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6051509Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6051737Z frame #6: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6051975Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6052201Z frame #8: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6052443Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x56166420fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6052680Z frame #10: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6053031Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x56166420dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6053265Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6053498Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6053726Z frame #14: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6053959Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6054183Z frame #16: <unknown function> + 0x147734 (0x561664226734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6054400Z frame #17: <unknown function> + 0x21ca82 (0x5616642fba82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6054624Z frame #18: <unknown function> + 0x21c9e4 (0x5616642fb9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6054842Z frame #19: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6055063Z frame #20: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6055067Z 
2025-07-17T11:07:25.6055157Z SIGABRT(6), PID: 1777266, Thread 1777356: 
2025-07-17T11:07:25.6055639Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6055855Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6056131Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6056336Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6056926Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6057227Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6057452Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6057456Z 
2025-07-17T11:07:25.6057548Z SIGABRT(6), PID: 1777272, Thread 1777739: 
2025-07-17T11:07:25.6058036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6058322Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6058526Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6058731Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6059237Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6059452Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6059661Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6059666Z 
2025-07-17T11:07:25.6059757Z SIGABRT(6), PID: 1777265, Thread 1777373: 
2025-07-17T11:07:25.6060229Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6060439Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6060646Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6060852Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6061438Z frame #4: <unknown function> + 0x3475ab (0x7f8ba8b0e5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6061649Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6061858Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6061863Z 
2025-07-17T11:07:25.6061960Z SIGABRT(6), PID: 1777265, Thread 1777789: 
2025-07-17T11:07:25.6062424Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6062878Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f8aba36908a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6063090Z frame #2: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6063294Z frame #3: pthread_kill + 0x12c (0x7f8babc789fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6063462Z frame #4: raise + 0x16 (0x7f8babc24476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6063628Z frame #5: abort + 0xd3 (0x7f8babc0a7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6063867Z frame #6: <unknown function> + 0x972cc (0x7f8a27ef22cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6064108Z frame #7: <unknown function> + 0x95f47 (0x7f8a27ef0f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6064333Z frame #8: <unknown function> + 0x34b71 (0x7f8a27e8fb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6064548Z frame #9: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6064829Z frame #10: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6064835Z 
2025-07-17T11:07:25.6064937Z SIGABRT(6), PID: 1777266, Thread 1777358: 
2025-07-17T11:07:25.6065493Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6065721Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6065930Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6066203Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6066705Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6066915Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6067127Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6067226Z SIGABRT(6), PID: 1777272, Thread 1777740: 
2025-07-17T11:07:25.6067690Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6067903Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6068110Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6068311Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6068811Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6069023Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6069232Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6069311Z 
2025-07-17T11:07:25.6069315Z 
2025-07-17T11:07:25.6069416Z SIGABRT(6), PID: 1777265, Thread 1780907: 
2025-07-17T11:07:25.6069877Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6070088Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6070261Z frame #2: __poll + 0x4f (0x7f8babcfabcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6070669Z frame #3: <unknown function> + 0x489f (0x7f8bab80689f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.6070895Z frame #4: <unknown function> + 0x1483bc (0x5587304fc3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6071148Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6071376Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6071617Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6071837Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6072083Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5587304e4a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6072312Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6072562Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5587304e2c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6072783Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6073088Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6073374Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6073613Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6073834Z frame #16: <unknown function> + 0x147734 (0x5587304fb734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6074057Z frame #17: <unknown function> + 0x21ca82 (0x5587305d0a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6074337Z frame #18: <unknown function> + 0x21c9e4 (0x5587305d09e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6074552Z frame #19: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6074763Z frame #20: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6074776Z 
2025-07-17T11:07:25.6074868Z SIGABRT(6), PID: 1777271, Thread 1781114: 
2025-07-17T11:07:25.6075343Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6075553Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6075760Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6075982Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f83309802dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6076436Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f826adbf048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6076858Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f826adc1bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6077095Z frame #6: <unknown function> + 0xdc253 (0x7f823d2e3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6077374Z frame #7: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6077587Z frame #8: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6077591Z 
2025-07-17T11:07:25.6077681Z SIGABRT(6), PID: 1777272, Thread 1777742: 
2025-07-17T11:07:25.6078153Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6078360Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6078566Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6078769Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6079275Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6079487Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6079698Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6079702Z 
2025-07-17T11:07:25.6079794Z SIGABRT(6), PID: 1777266, Thread 1777361: 
2025-07-17T11:07:25.6080270Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6080477Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6080690Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6080957Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6081528Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6081742Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6081959Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6081963Z 
2025-07-17T11:07:25.6082050Z SIGABRT(6), PID: 1777272, Thread 1777745: 
2025-07-17T11:07:25.6082582Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6082786Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6082993Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6083192Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6083692Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6083898Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6084108Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6084113Z 
2025-07-17T11:07:25.6084205Z SIGABRT(6), PID: 1777266, Thread 1777363: 
2025-07-17T11:07:25.6084670Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6084877Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6085091Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6085362Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6085862Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6086080Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6086286Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6086292Z 
2025-07-17T11:07:25.6086392Z SIGABRT(6), PID: 1777271, Thread 1781115: 
2025-07-17T11:07:25.6086855Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6087062Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6087264Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6087491Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f83309802dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6087961Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f826adb9741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6088194Z frame #5: <unknown function> + 0xdc253 (0x7f823d2e3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6088398Z frame #6: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6088610Z frame #7: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6088614Z 
2025-07-17T11:07:25.6088705Z SIGABRT(6), PID: 1777272, Thread 1777747: 
2025-07-17T11:07:25.6089242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6089513Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6089725Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6089924Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6090420Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6090694Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6090906Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6090911Z 
2025-07-17T11:07:25.6091002Z SIGABRT(6), PID: 1777271, Thread 1781118: 
2025-07-17T11:07:25.6091476Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6091681Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6091847Z frame #2: ioctl + 0x3f (0x7f8330a0694f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6092082Z frame #3: <unknown function> + 0x13eef0 (0x7f81acca7ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6092317Z frame #4: <unknown function> + 0x1378c4 (0x7f81acca08c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6092545Z frame #5: <unknown function> + 0x95b2b (0x7f81acbfeb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6092772Z frame #6: <unknown function> + 0x34b71 (0x7f81acb9db71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6092978Z frame #7: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6093192Z frame #8: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6093352Z SIGABRT(6), PID: 1777266, Thread 1777365: 
2025-07-17T11:07:25.6093822Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6094030Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6094239Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6094439Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6094936Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6095146Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6095365Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6095372Z 
2025-07-17T11:07:25.6095376Z 
2025-07-17T11:07:25.6095467Z SIGABRT(6), PID: 1777272, Thread 1777749: 
2025-07-17T11:07:25.6096002Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6096208Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6096415Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6096610Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6097208Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6097418Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6097706Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6097711Z 
2025-07-17T11:07:25.6097803Z SIGABRT(6), PID: 1777266, Thread 1777366: 
2025-07-17T11:07:25.6098274Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6098557Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6098764Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6098969Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6099462Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6099678Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6099886Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6099890Z 
2025-07-17T11:07:25.6099985Z SIGABRT(6), PID: 1777271, Thread 1781539: 
2025-07-17T11:07:25.6100451Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6100660Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6100826Z frame #2: __poll + 0x4f (0x7f8330a04bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6101041Z frame #3: <unknown function> + 0xd4540 (0x7f81f4e17540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6101248Z frame #4: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6101551Z frame #5: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6101555Z 
2025-07-17T11:07:25.6101650Z SIGABRT(6), PID: 1777271, Thread 1781547: 
2025-07-17T11:07:25.6102122Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6102335Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6102507Z frame #2: __poll + 0x4f (0x7f8330a04bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6102729Z frame #3: <unknown function> + 0xd65a2 (0x7f81f4e195a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6102939Z frame #4: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6103146Z frame #5: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6103152Z 
2025-07-17T11:07:25.6103253Z SIGABRT(6), PID: 1777272, Thread 1777750: 
2025-07-17T11:07:25.6103723Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6103935Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6104137Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6104348Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6104844Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6105123Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6105339Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6105345Z 
2025-07-17T11:07:25.6105504Z SIGABRT(6), PID: 1777266, Thread 1777368: 
2025-07-17T11:07:25.6105979Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6106193Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6106465Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6106674Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6107171Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6107385Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6107597Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6107601Z 
2025-07-17T11:07:25.6107699Z SIGABRT(6), PID: 1777271, Thread 1781556: 
2025-07-17T11:07:25.6108163Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6108374Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6108578Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6108778Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6108991Z frame #4: <unknown function> + 0xd1e7f (0x7f81f4e14e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6109198Z frame #5: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6109472Z frame #6: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6109482Z 
2025-07-17T11:07:25.6109573Z SIGABRT(6), PID: 1777272, Thread 1777751: 
2025-07-17T11:07:25.6110051Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6110258Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6110467Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6110662Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6111162Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6111366Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6111581Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6111585Z 
2025-07-17T11:07:25.6111673Z SIGABRT(6), PID: 1777266, Thread 1777370: 
2025-07-17T11:07:25.6112137Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6112345Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6112555Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6112750Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6113317Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6113591Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6113807Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6113811Z 
2025-07-17T11:07:25.6113902Z SIGABRT(6), PID: 1777265, Thread 1780908: 
2025-07-17T11:07:25.6114372Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6114646Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6114833Z frame #2: epoll_wait + 0x5e (0x7f8babd07e2e in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6115170Z frame #3: uv__io_poll + 0x6bd (0x7f8afc8bf44d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6115492Z frame #4: uv_run + 0x12a (0x7f8afc8b336a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6115869Z frame #5: <unknown function> + 0x658a08a (0x7f8af0c2708a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6116108Z frame #6: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6116319Z frame #7: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6116539Z frame #8: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6116543Z 
2025-07-17T11:07:25.6116633Z SIGABRT(6), PID: 1777271, Thread 1781567: 
2025-07-17T11:07:25.6117104Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6117313Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6117725Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6117934Z frame #3: <unknown function> + 0x9ca53 (0x7f8330988a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6118191Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5616641f3bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6118419Z frame #5: <unknown function> + 0x1d314f (0x5616642b214f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6118644Z frame #6: <unknown function> + 0x1d2f6b (0x5616642b1f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6118864Z frame #7: <unknown function> + 0x6df36 (0x56166414cf36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6119110Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6119334Z frame #9: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6119579Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6119806Z frame #11: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6120045Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6120268Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6120507Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6120733Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6120963Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6121249Z frame #17: <unknown function> + 0x147734 (0x561664226734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6121538Z frame #18: <unknown function> + 0x21ca82 (0x5616642fba82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6121764Z frame #19: <unknown function> + 0x21c9e4 (0x5616642fb9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6121983Z frame #20: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6122193Z frame #21: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6122260Z 
2025-07-17T11:07:25.6122360Z SIGABRT(6), PID: 1777272, Thread 1777752: 
2025-07-17T11:07:25.6122828Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6123045Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6123251Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6123460Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6123959Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6124171Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6124378Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6124382Z 
2025-07-17T11:07:25.6124476Z SIGABRT(6), PID: 1777266, Thread 1777372: 
2025-07-17T11:07:25.6124949Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6125162Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6125370Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6125668Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6126172Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6126388Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6126602Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6126606Z 
2025-07-17T11:07:25.6126697Z SIGABRT(6), PID: 1777271, Thread 1781583: 
2025-07-17T11:07:25.6127167Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6127371Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6127543Z frame #2: read + 0x4c (0x7f8330a0081c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6127719Z frame #3: _Py_read + 0x4c (0x5616642b9a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6127946Z frame #4: <unknown function> + 0x208c3b (0x5616642e7c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6128162Z frame #5: <unknown function> + 0x14648d (0x56166422548d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6128408Z frame #6: PyObject_VectorcallMethod + 0x85 (0x561664234215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6128626Z frame #7: <unknown function> + 0x208ae7 (0x5616642e7ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6128844Z frame #8: <unknown function> + 0x208472 (0x5616642e7472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6129137Z frame #9: <unknown function> + 0x1da6ee (0x5616642b96ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6129369Z frame #10: <unknown function> + 0x1483bc (0x5616642273bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6129676Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6129905Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6130140Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x56166420b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6130450Z frame #14: <unknown function> + 0x147734 (0x561664226734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6130700Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x56166420dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6130928Z frame #16: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6131162Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6131387Z frame #18: _PyFunction_Vectorcall + 0x6c (0x56166421b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6131622Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x56166420b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6131846Z frame #20: <unknown function> + 0x147734 (0x561664226734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6132068Z frame #21: <unknown function> + 0x21ca82 (0x5616642fba82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6132291Z frame #22: <unknown function> + 0x21c9e4 (0x5616642fb9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6132505Z frame #23: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6132722Z frame #24: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6132818Z SIGABRT(6), PID: 1777272, Thread 1777753: 
2025-07-17T11:07:25.6133298Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6133580Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6133794Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6133997Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6134497Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6134703Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6134916Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6134920Z 
2025-07-17T11:07:25.6134923Z 
2025-07-17T11:07:25.6135017Z SIGABRT(6), PID: 1777266, Thread 1777374: 
2025-07-17T11:07:25.6135488Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6135698Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6135959Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6136164Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6136667Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6136874Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6137168Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6137173Z 
2025-07-17T11:07:25.6137269Z SIGABRT(6), PID: 1777272, Thread 1777754: 
2025-07-17T11:07:25.6137816Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6138025Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6138235Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6138514Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6139014Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6139222Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6139436Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6139442Z 
2025-07-17T11:07:25.6139536Z SIGABRT(6), PID: 1777265, Thread 1781526: 
2025-07-17T11:07:25.6140011Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6140221Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6140433Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6140658Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f8babc762dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6141107Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f8ae60b1048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6141532Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f8ae60b3bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6141864Z frame #6: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6142074Z frame #7: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6142290Z frame #8: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6142295Z 
2025-07-17T11:07:25.6142391Z SIGABRT(6), PID: 1777266, Thread 1777376: 
2025-07-17T11:07:25.6142864Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6143079Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6143282Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6143488Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6143991Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6144209Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6144415Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6144513Z SIGABRT(6), PID: 1777272, Thread 1777755: 
2025-07-17T11:07:25.6144977Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6145185Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6145456Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6145665Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6146222Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6146437Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6146646Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6146711Z 
2025-07-17T11:07:25.6146714Z 
2025-07-17T11:07:25.6146812Z SIGABRT(6), PID: 1777272, Thread 1777756: 
2025-07-17T11:07:25.6147277Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6147490Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6147696Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6147901Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6148397Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6148609Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6148821Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6148920Z SIGABRT(6), PID: 1777266, Thread 1777378: 
2025-07-17T11:07:25.6149387Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6149600Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6149880Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6150089Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6150584Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6150800Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6151009Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6151013Z 
2025-07-17T11:07:25.6151017Z 
2025-07-17T11:07:25.6151111Z SIGABRT(6), PID: 1777265, Thread 1781527: 
2025-07-17T11:07:25.6151577Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6151795Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6151997Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6152227Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f8babc762dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6152701Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f8ae60ab741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6152941Z frame #5: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6153150Z frame #6: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6153364Z frame #7: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6153368Z 
2025-07-17T11:07:25.6153527Z SIGABRT(6), PID: 1777265, Thread 1781529: 
2025-07-17T11:07:25.6154058Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6154274Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6154448Z frame #2: ioctl + 0x3f (0x7f8babcfc94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6154689Z frame #3: <unknown function> + 0x13eef0 (0x7f8a27f99ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6155000Z frame #4: <unknown function> + 0x1378c4 (0x7f8a27f928c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6155236Z frame #5: <unknown function> + 0x95b2b (0x7f8a27ef0b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6155465Z frame #6: <unknown function> + 0x34b71 (0x7f8a27e8fb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6155679Z frame #7: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6155895Z frame #8: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6155899Z 
2025-07-17T11:07:25.6155995Z SIGABRT(6), PID: 1777272, Thread 1777757: 
2025-07-17T11:07:25.6156461Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6156672Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6156881Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6157081Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6157577Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6157786Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6158064Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6158070Z 
2025-07-17T11:07:25.6158167Z SIGABRT(6), PID: 1777266, Thread 1777379: 
2025-07-17T11:07:25.6158635Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6158850Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6159055Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6159255Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6159749Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6159966Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6160175Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6160179Z 
2025-07-17T11:07:25.6160275Z SIGABRT(6), PID: 1777272, Thread 1777758: 
2025-07-17T11:07:25.6160738Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6160950Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6161155Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6161353Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6161975Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6162200Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6162412Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6162510Z SIGABRT(6), PID: 1777265, Thread 1781545: 
2025-07-17T11:07:25.6162976Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6163252Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6163422Z frame #2: __poll + 0x4f (0x7f8babcfabcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6163641Z frame #3: <unknown function> + 0xd4540 (0x7f8a70109540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6163847Z frame #4: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6164067Z frame #5: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6164071Z 
2025-07-17T11:07:25.6164074Z 
2025-07-17T11:07:25.6164166Z SIGABRT(6), PID: 1777266, Thread 1777381: 
2025-07-17T11:07:25.6164636Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6164847Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6165056Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6165255Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6165755Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6166038Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6166253Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6166257Z 
2025-07-17T11:07:25.6166352Z SIGABRT(6), PID: 1777265, Thread 1781553: 
2025-07-17T11:07:25.6166819Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6167034Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6167201Z frame #2: __poll + 0x4f (0x7f8babcfabcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6167421Z frame #3: <unknown function> + 0xd65a2 (0x7f8a7010b5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6167630Z frame #4: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6167848Z frame #5: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6167852Z 
2025-07-17T11:07:25.6167945Z SIGABRT(6), PID: 1777272, Thread 1777759: 
2025-07-17T11:07:25.6168409Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6168616Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6168823Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6169018Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6169586Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6169857Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6170075Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6170166Z SIGABRT(6), PID: 1777266, Thread 1777383: 
2025-07-17T11:07:25.6170640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6170911Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6171123Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6171322Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6171822Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6172035Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6172248Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6172251Z 
2025-07-17T11:07:25.6172254Z 
2025-07-17T11:07:25.6172346Z SIGABRT(6), PID: 1777265, Thread 1781555: 
2025-07-17T11:07:25.6172819Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6173027Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6173236Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6173431Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6173649Z frame #4: <unknown function> + 0xd1e7f (0x7f8a70106e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6173925Z frame #5: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6174138Z frame #6: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6174142Z 
2025-07-17T11:07:25.6174232Z SIGABRT(6), PID: 1777271, Thread 1781627: 
2025-07-17T11:07:25.6174704Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6174909Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6175119Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6175315Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6175701Z frame #4: <unknown function> + 0x5a02023 (0x7f8274dad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6176317Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8274db11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6176918Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8274da81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6177304Z frame #7: <unknown function> + 0x79d655 (0x7f8286298655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6177533Z frame #8: <unknown function> + 0xdc253 (0x7f823d2e3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6177837Z frame #9: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6178059Z frame #10: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6178137Z 
2025-07-17T11:07:25.6178241Z SIGABRT(6), PID: 1777266, Thread 1777385: 
2025-07-17T11:07:25.6178713Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6178927Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6179207Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6179414Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6179915Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6180131Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6180343Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6180441Z SIGABRT(6), PID: 1777272, Thread 1777760: 
2025-07-17T11:07:25.6180910Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6181115Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6181320Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6181522Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6182016Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6182229Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6182526Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6182530Z 
2025-07-17T11:07:25.6182533Z 
2025-07-17T11:07:25.6182627Z SIGABRT(6), PID: 1777265, Thread 1781563: 
2025-07-17T11:07:25.6183094Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6183310Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6183517Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6183729Z frame #3: <unknown function> + 0x9ca53 (0x7f8babc7ea53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6183982Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5587304c8bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6184216Z frame #5: <unknown function> + 0x1d314f (0x55873058714f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6184440Z frame #6: <unknown function> + 0x1d2f6b (0x558730586f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6184663Z frame #7: <unknown function> + 0x6df36 (0x558730421f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6184901Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6185129Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6185368Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6185595Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6185903Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6186221Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6186455Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6186679Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6186908Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6187201Z frame #17: <unknown function> + 0x147734 (0x5587304fb734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6187418Z frame #18: <unknown function> + 0x21ca82 (0x5587305d0a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6187641Z frame #19: <unknown function> + 0x21c9e4 (0x5587305d09e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6187857Z frame #20: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6188076Z frame #21: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6188083Z 
2025-07-17T11:07:25.6188175Z SIGABRT(6), PID: 1777266, Thread 1777387: 
2025-07-17T11:07:25.6188655Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6188866Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6189080Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6189281Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6189785Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6189995Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6190289Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6190293Z 
2025-07-17T11:07:25.6190382Z SIGABRT(6), PID: 1777272, Thread 1777761: 
2025-07-17T11:07:25.6190856Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6191061Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6191273Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6191469Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6191974Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6192181Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6192391Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6192395Z 
2025-07-17T11:07:25.6192490Z SIGABRT(6), PID: 1777265, Thread 1781572: 
2025-07-17T11:07:25.6192963Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6193176Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6193342Z frame #2: read + 0x4c (0x7f8babcf681c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6193523Z frame #3: _Py_read + 0x4c (0x55873058ea6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6193824Z frame #4: <unknown function> + 0x208c3b (0x5587305bcc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6194052Z frame #5: <unknown function> + 0x14648d (0x5587304fa48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6194358Z frame #6: PyObject_VectorcallMethod + 0x85 (0x558730509215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6194592Z frame #7: <unknown function> + 0x208ae7 (0x5587305bcae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6194809Z frame #8: <unknown function> + 0x208472 (0x5587305bc472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6195092Z frame #9: <unknown function> + 0x1da6ee (0x55873058e6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6195317Z frame #10: <unknown function> + 0x1483bc (0x5587304fc3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6195562Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6195788Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6196028Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5587304e03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6196249Z frame #14: <unknown function> + 0x147734 (0x5587304fb734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6196507Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5587304e2c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6196732Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6196972Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6197191Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5587304f00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6197430Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5587304e07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6197653Z frame #20: <unknown function> + 0x147734 (0x5587304fb734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6197881Z frame #21: <unknown function> + 0x21ca82 (0x5587305d0a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6198167Z frame #22: <unknown function> + 0x21c9e4 (0x5587305d09e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6198389Z frame #23: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6198602Z frame #24: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6198608Z 
2025-07-17T11:07:25.6198706Z SIGABRT(6), PID: 1777272, Thread 1777762: 
2025-07-17T11:07:25.6199177Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6199391Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6199599Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6199801Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6200302Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6200517Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6200730Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6200736Z 
2025-07-17T11:07:25.6200832Z SIGABRT(6), PID: 1777266, Thread 1777389: 
2025-07-17T11:07:25.6201297Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6201508Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6201783Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6202056Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6202561Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6202786Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6203073Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6203078Z 
2025-07-17T11:07:25.6203177Z SIGABRT(6), PID: 1777272, Thread 1777763: 
2025-07-17T11:07:25.6203650Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6203865Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6204080Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6204288Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6204792Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6205007Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6205219Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6205223Z 
2025-07-17T11:07:25.6205323Z SIGABRT(6), PID: 1777266, Thread 1777391: 
2025-07-17T11:07:25.6205792Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6206010Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6206288Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6206503Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6207003Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6207220Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6207428Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6207432Z 
2025-07-17T11:07:25.6207529Z SIGABRT(6), PID: 1777272, Thread 1777764: 
2025-07-17T11:07:25.6208004Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6208221Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6208426Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6208631Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6209121Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6209332Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6209546Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6209550Z 
2025-07-17T11:07:25.6209642Z SIGABRT(6), PID: 1777266, Thread 1777394: 
2025-07-17T11:07:25.6210192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6210482Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6210696Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6210899Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6211463Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6211673Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6211885Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6211892Z 
2025-07-17T11:07:25.6211980Z SIGABRT(6), PID: 1777272, Thread 1777765: 
2025-07-17T11:07:25.6212457Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6212662Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6212872Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6213067Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6213569Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6213776Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6213992Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6213996Z 
2025-07-17T11:07:25.6214085Z SIGABRT(6), PID: 1777266, Thread 1777397: 
2025-07-17T11:07:25.6214641Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6214855Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6215073Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6215277Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6215780Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6216045Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6216265Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6216271Z 
2025-07-17T11:07:25.6216366Z SIGABRT(6), PID: 1777272, Thread 1777766: 
2025-07-17T11:07:25.6216843Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6217046Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6217257Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6217454Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6217965Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6218286Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6218580Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6218588Z 
2025-07-17T11:07:25.6218683Z SIGABRT(6), PID: 1777266, Thread 1777399: 
2025-07-17T11:07:25.6219156Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6219368Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6219663Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6219863Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6220360Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6220578Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6220792Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6220891Z SIGABRT(6), PID: 1777271, Thread 1781628: 
2025-07-17T11:07:25.6221358Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6221573Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6221774Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6222033Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6222615Z frame #4: <unknown function> + 0x5a02023 (0x7f8274dad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6223226Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8274db11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6224147Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8274da81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6224573Z frame #7: <unknown function> + 0x79d655 (0x7f8286298655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6224880Z frame #8: <unknown function> + 0xdc253 (0x7f823d2e3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6225204Z frame #9: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6225531Z frame #10: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6225538Z 
2025-07-17T11:07:25.6225542Z 
2025-07-17T11:07:25.6225709Z SIGABRT(6), PID: 1777272, Thread 1777767: 
2025-07-17T11:07:25.6226278Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6226526Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6226858Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6227119Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6227741Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6227994Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6228361Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6228368Z 
2025-07-17T11:07:25.6228658Z SIGABRT(6), PID: 1777266, Thread 1777402: 
2025-07-17T11:07:25.6229243Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6229556Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6229882Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6230166Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6230693Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6231233Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6231491Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6231673Z SIGABRT(6), PID: 1777272, Thread 1777768: 
2025-07-17T11:07:25.6232184Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6232455Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6232786Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6233088Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6233633Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6233928Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6234260Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6234265Z 
2025-07-17T11:07:25.6234268Z 
2025-07-17T11:07:25.6234518Z SIGABRT(6), PID: 1777272, Thread 1777769: 
2025-07-17T11:07:25.6235053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6235353Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6235595Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6235899Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6236417Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6236794Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6237045Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6237225Z SIGABRT(6), PID: 1777266, Thread 1777405: 
2025-07-17T11:07:25.6237732Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6238027Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6238333Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6238763Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6239372Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6239769Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6240028Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6240058Z 
2025-07-17T11:07:25.6240062Z 
2025-07-17T11:07:25.6240261Z SIGABRT(6), PID: 1777266, Thread 1777408: 
2025-07-17T11:07:25.6240911Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6241183Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6241475Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6241723Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6242357Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6242629Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6242943Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6242949Z 
2025-07-17T11:07:25.6243082Z SIGABRT(6), PID: 1777272, Thread 1777770: 
2025-07-17T11:07:25.6243631Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6243861Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6244236Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6244502Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6245161Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6245416Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6245811Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6245817Z 
2025-07-17T11:07:25.6246012Z SIGABRT(6), PID: 1777266, Thread 1777411: 
2025-07-17T11:07:25.6246606Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6246852Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6247145Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6247390Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6248010Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6248345Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6248603Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6248608Z 
2025-07-17T11:07:25.6248786Z SIGABRT(6), PID: 1777265, Thread 1781587: 
2025-07-17T11:07:25.6249296Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6249729Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6250069Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6250363Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6250785Z frame #4: <unknown function> + 0x5a02023 (0x7f8af009f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6251432Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8af00a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6252130Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8af009a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6252694Z frame #7: <unknown function> + 0x79d655 (0x7f8b0158a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6252980Z frame #8: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6253390Z frame #9: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6253647Z frame #10: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6253652Z 
2025-07-17T11:07:25.6253828Z SIGABRT(6), PID: 1777272, Thread 1777771: 
2025-07-17T11:07:25.6254426Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6254741Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6254986Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6255290Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6256006Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6256356Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6256634Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6256685Z 
2025-07-17T11:07:25.6256818Z SIGABRT(6), PID: 1777266, Thread 1777414: 
2025-07-17T11:07:25.6257423Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6257679Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6258025Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6258285Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6258875Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6259147Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6259441Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6259448Z 
2025-07-17T11:07:25.6259561Z SIGABRT(6), PID: 1777272, Thread 1777772: 
2025-07-17T11:07:25.6260187Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6260439Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6260972Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6261306Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6261874Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6262194Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6262624Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6262629Z 
2025-07-17T11:07:25.6262766Z SIGABRT(6), PID: 1777272, Thread 1777773: 
2025-07-17T11:07:25.6263317Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6263567Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6263912Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6264191Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6264776Z frame #4: <unknown function> + 0x3475ab (0x7f1906d5f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6265032Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6265330Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6265334Z 
2025-07-17T11:07:25.6265446Z SIGABRT(6), PID: 1777266, Thread 1777417: 
2025-07-17T11:07:25.6266102Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6266354Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6266746Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6266987Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6267555Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6268072Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6268329Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6268334Z 
2025-07-17T11:07:25.6268511Z SIGABRT(6), PID: 1777271, Thread 1781629: 
2025-07-17T11:07:25.6269023Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6269315Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6269626Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6269929Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6270344Z frame #4: <unknown function> + 0x5a02023 (0x7f8274dad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6270988Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8274db11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6271713Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8274da81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6272293Z frame #7: <unknown function> + 0x79d655 (0x7f8286298655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6272596Z frame #8: <unknown function> + 0xdc253 (0x7f823d2e3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6272899Z frame #9: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6273244Z frame #10: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6273251Z 
2025-07-17T11:07:25.6273428Z SIGABRT(6), PID: 1777272, Thread 1777797: 
2025-07-17T11:07:25.6273919Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6274549Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f18185be08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6274805Z frame #2: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6275107Z frame #3: pthread_kill + 0x12c (0x7f1909ec99fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6275313Z frame #4: raise + 0x16 (0x7f1909e75476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6275658Z frame #5: abort + 0xd3 (0x7f1909e5b7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6275995Z frame #6: <unknown function> + 0x972cc (0x7f17861472cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6276326Z frame #7: <unknown function> + 0x95f47 (0x7f1786145f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6276613Z frame #8: <unknown function> + 0x34b71 (0x7f17860e4b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6276905Z frame #9: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6277161Z frame #10: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6277266Z 
2025-07-17T11:07:25.6277468Z SIGABRT(6), PID: 1777266, Thread 1777420: 
2025-07-17T11:07:25.6278077Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6278332Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6278625Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6278870Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6279497Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6279774Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6280093Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6280097Z 
2025-07-17T11:07:25.6280229Z SIGABRT(6), PID: 1777272, Thread 1780511: 
2025-07-17T11:07:25.6280783Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6281011Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6281367Z frame #2: __poll + 0x4f (0x7f1909f4bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6281815Z frame #3: <unknown function> + 0x489f (0x7f1909a5789f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.6282129Z frame #4: <unknown function> + 0x1483bc (0x56197ceba3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6282611Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6282985Z frame #6: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6283359Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6283685Z frame #8: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6284044Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x56197cea2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6284358Z frame #10: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6284651Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x56197cea0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6285024Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6285324Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6285638Z frame #14: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6285915Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6286245Z frame #16: <unknown function> + 0x147734 (0x56197ceb9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6286487Z frame #17: <unknown function> + 0x21ca82 (0x56197cf8ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6286881Z frame #18: <unknown function> + 0x21c9e4 (0x56197cf8e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6287139Z frame #19: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6287431Z frame #20: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6287436Z 
2025-07-17T11:07:25.6287574Z SIGABRT(6), PID: 1777266, Thread 1777423: 
2025-07-17T11:07:25.6288216Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6288536Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6288853Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6289207Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6289817Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6290067Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6290408Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6290413Z 
2025-07-17T11:07:25.6290566Z SIGABRT(6), PID: 1777266, Thread 1777426: 
2025-07-17T11:07:25.6291130Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6291440Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6291685Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6292012Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6292572Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6292867Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6293212Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6293220Z 
2025-07-17T11:07:25.6293465Z SIGABRT(6), PID: 1777266, Thread 1777430: 
2025-07-17T11:07:25.6293955Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6294334Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6294679Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6294964Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6295497Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6295771Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6296165Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6296172Z 
2025-07-17T11:07:25.6296509Z SIGABRT(6), PID: 1777265, Thread 1781588: 
2025-07-17T11:07:25.6297023Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6297318Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6297567Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6297895Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6298346Z frame #4: <unknown function> + 0x5a02023 (0x7f8af009f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6299005Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8af00a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6299742Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8af009a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6300214Z frame #7: <unknown function> + 0x79d655 (0x7f8b0158a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6300607Z frame #8: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6300880Z frame #9: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6301180Z frame #10: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6301187Z 
2025-07-17T11:07:25.6301322Z SIGABRT(6), PID: 1777272, Thread 1781105: 
2025-07-17T11:07:25.6301883Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6302114Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6302509Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6302777Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f1909ec72dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6303300Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f1844306048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6303760Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f1844308bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6304161Z frame #6: <unknown function> + 0xdc253 (0x7f181682a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6304676Z frame #7: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6305006Z frame #8: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6305011Z 
2025-07-17T11:07:25.6305148Z SIGABRT(6), PID: 1777266, Thread 1777433: 
2025-07-17T11:07:25.6305742Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6306081Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6306424Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6306696Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6307294Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6307574Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6307871Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6307876Z 
2025-07-17T11:07:25.6307987Z SIGABRT(6), PID: 1777266, Thread 1777436: 
2025-07-17T11:07:25.6308628Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6308878Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6309198Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6309441Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6310082Z frame #4: <unknown function> + 0x3475ab (0x7fa32bd445ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6310468Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6310726Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6310733Z 
2025-07-17T11:07:25.6310929Z SIGABRT(6), PID: 1777266, Thread 1777795: 
2025-07-17T11:07:25.6311564Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6312081Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fa23d5a008a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6312392Z frame #2: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6312718Z frame #3: pthread_kill + 0x12c (0x7fa32eeae9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6312927Z frame #4: raise + 0x16 (0x7fa32ee5a476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6313175Z frame #5: abort + 0xd3 (0x7fa32ee407f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6313451Z frame #6: <unknown function> + 0x972cc (0x7fa1ab1292cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6313819Z frame #7: <unknown function> + 0x95f47 (0x7fa1ab127f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6314109Z frame #8: <unknown function> + 0x34b71 (0x7fa1ab0c6b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6314424Z frame #9: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6314682Z frame #10: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6314779Z 
2025-07-17T11:07:25.6314969Z SIGABRT(6), PID: 1777272, Thread 1781106: 
2025-07-17T11:07:25.6315532Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6315950Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6316206Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6316588Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f1909ec72dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6317107Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f1844300741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6317401Z frame #5: <unknown function> + 0xdc253 (0x7f181682a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6317746Z frame #6: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6318068Z frame #7: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6318073Z 
2025-07-17T11:07:25.6318320Z SIGABRT(6), PID: 1777272, Thread 1781107: 
2025-07-17T11:07:25.6318875Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6319125Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6319443Z frame #2: ioctl + 0x3f (0x7f1909f4d94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6319737Z frame #3: <unknown function> + 0x13eef0 (0x7f17861eeef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6320052Z frame #4: <unknown function> + 0x1378c4 (0x7f17861e78c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6320329Z frame #5: <unknown function> + 0x95b2b (0x7f1786145b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6320745Z frame #6: <unknown function> + 0x34b71 (0x7f17860e4b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6320980Z frame #7: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6321374Z frame #8: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6321379Z 
2025-07-17T11:07:25.6321514Z SIGABRT(6), PID: 1777271, Thread 1781630: 
2025-07-17T11:07:25.6322066Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6322381Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6322608Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6322978Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6323407Z frame #4: <unknown function> + 0x5a02023 (0x7f8274dad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6324054Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8274db11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6324705Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8274da81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6325150Z frame #7: <unknown function> + 0x79d655 (0x7f8286298655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6325521Z frame #8: <unknown function> + 0xdc253 (0x7f823d2e3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6326011Z frame #9: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6326354Z frame #10: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6326359Z 
2025-07-17T11:07:25.6326543Z SIGABRT(6), PID: 1777266, Thread 1781523: 
2025-07-17T11:07:25.6327059Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6327469Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6327706Z frame #2: __poll + 0x4f (0x7fa32ef30bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6328223Z frame #3: <unknown function> + 0x489f (0x7fa32ea3c89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.6328492Z frame #4: <unknown function> + 0x1483bc (0x55b74690b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6328823Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6329076Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6329480Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6329769Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6330104Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55b7468f3a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6330374Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6330687Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b7468f1c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6331019Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6331384Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6331739Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6332057Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6332317Z frame #16: <unknown function> + 0x147734 (0x55b74690a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6332808Z frame #17: <unknown function> + 0x21ca82 (0x55b7469dfa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6333113Z frame #18: <unknown function> + 0x21c9e4 (0x55b7469df9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6333419Z frame #19: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6333675Z frame #20: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6333681Z 
2025-07-17T11:07:25.6333861Z SIGABRT(6), PID: 1777272, Thread 1781544: 
2025-07-17T11:07:25.6334358Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6334759Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6334966Z frame #2: __poll + 0x4f (0x7f1909f4bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6335270Z frame #3: <unknown function> + 0xd4540 (0x7f17ce35e540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6335516Z frame #4: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6335792Z frame #5: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6335796Z 
2025-07-17T11:07:25.6336125Z SIGABRT(6), PID: 1777272, Thread 1781552: 
2025-07-17T11:07:25.6336748Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6337127Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6337335Z frame #2: __poll + 0x4f (0x7f1909f4bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6337640Z frame #3: <unknown function> + 0xd65a2 (0x7f17ce3605a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6337960Z frame #4: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6338367Z frame #5: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6338372Z 
2025-07-17T11:07:25.6338511Z SIGABRT(6), PID: 1777272, Thread 1781558: 
2025-07-17T11:07:25.6339090Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6339474Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6339815Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6340082Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6340384Z frame #4: <unknown function> + 0xd1e7f (0x7f17ce35be7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6340654Z frame #5: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6340945Z frame #6: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6340949Z 
2025-07-17T11:07:25.6341061Z SIGABRT(6), PID: 1777266, Thread 1781524: 
2025-07-17T11:07:25.6341690Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6341942Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6342359Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6342628Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa32eeac2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6343111Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fa2692e8048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6348931Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fa2692eabf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6349230Z frame #6: <unknown function> + 0xdc253 (0x7fa23b80c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6349472Z frame #7: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6349700Z frame #8: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6349708Z 
2025-07-17T11:07:25.6349816Z SIGABRT(6), PID: 1777272, Thread 1781564: 
2025-07-17T11:07:25.6350318Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6350535Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6350743Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6350955Z frame #3: <unknown function> + 0x9ca53 (0x7f1909ecfa53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6351216Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x56197ce86bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6351450Z frame #5: <unknown function> + 0x1d314f (0x56197cf4514f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6351787Z frame #6: <unknown function> + 0x1d2f6b (0x56197cf44f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6352089Z frame #7: <unknown function> + 0x6df36 (0x56197cddff36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6352338Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6352569Z frame #9: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6352814Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6353120Z frame #11: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6353356Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6353589Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6353821Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6354051Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6354281Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6354504Z frame #17: <unknown function> + 0x147734 (0x56197ceb9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6354721Z frame #18: <unknown function> + 0x21ca82 (0x56197cf8ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6354944Z frame #19: <unknown function> + 0x21c9e4 (0x56197cf8e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6355161Z frame #20: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6355376Z frame #21: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6355381Z 
2025-07-17T11:07:25.6355479Z SIGABRT(6), PID: 1777265, Thread 1781589: 
2025-07-17T11:07:25.6355973Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6356256Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6356468Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6356676Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6357060Z frame #4: <unknown function> + 0x5a02023 (0x7f8af009f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6357619Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8af00a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6358214Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8af009a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6358600Z frame #7: <unknown function> + 0x79d655 (0x7f8b0158a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6358827Z frame #8: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6359039Z frame #9: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6359255Z frame #10: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6359259Z 
2025-07-17T11:07:25.6359357Z SIGABRT(6), PID: 1777266, Thread 1781525: 
2025-07-17T11:07:25.6359903Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6360120Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6360391Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6360625Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa32eeac2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6361103Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fa2692e2741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6361403Z frame #5: <unknown function> + 0xdc253 (0x7fa23b80c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6361612Z frame #6: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6361824Z frame #7: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6361918Z SIGABRT(6), PID: 1777272, Thread 1781574: 
2025-07-17T11:07:25.6362398Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6362606Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6362776Z frame #2: read + 0x4c (0x7f1909f4781c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6362957Z frame #3: _Py_read + 0x4c (0x56197cf4ca6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6363187Z frame #4: <unknown function> + 0x208c3b (0x56197cf7ac3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6363406Z frame #5: <unknown function> + 0x14648d (0x56197ceb848d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6363656Z frame #6: PyObject_VectorcallMethod + 0x85 (0x56197cec7215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6363880Z frame #7: <unknown function> + 0x208ae7 (0x56197cf7aae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6364105Z frame #8: <unknown function> + 0x208472 (0x56197cf7a472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6364394Z frame #9: <unknown function> + 0x1da6ee (0x56197cf4c6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6364625Z frame #10: <unknown function> + 0x1483bc (0x56197ceba3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6364870Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6365098Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6365333Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x56197ce9e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6365555Z frame #14: <unknown function> + 0x147734 (0x56197ceb9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6365805Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x56197cea0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6366037Z frame #16: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6366275Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6366499Z frame #18: _PyFunction_Vectorcall + 0x6c (0x56197ceae0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6366729Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x56197ce9e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6366951Z frame #20: <unknown function> + 0x147734 (0x56197ceb9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6367168Z frame #21: <unknown function> + 0x21ca82 (0x56197cf8ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6367387Z frame #22: <unknown function> + 0x21c9e4 (0x56197cf8e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6367600Z frame #23: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6367906Z frame #24: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6367913Z 
2025-07-17T11:07:25.6367917Z 
2025-07-17T11:07:25.6368075Z SIGABRT(6), PID: 1777266, Thread 1781528: 
2025-07-17T11:07:25.6368560Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6368774Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6369011Z frame #2: ioctl + 0x3f (0x7fa32ef3294f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6369248Z frame #3: <unknown function> + 0x13eef0 (0x7fa1ab1d0ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6369488Z frame #4: <unknown function> + 0x1378c4 (0x7fa1ab1c98c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6369726Z frame #5: <unknown function> + 0x95b2b (0x7fa1ab127b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6369960Z frame #6: <unknown function> + 0x34b71 (0x7fa1ab0c6b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6370181Z frame #7: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6370397Z frame #8: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6370401Z 
2025-07-17T11:07:25.6370496Z SIGABRT(6), PID: 1777271, Thread 1781632: 
2025-07-17T11:07:25.6370991Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6371203Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6371410Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6371616Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6371998Z frame #4: <unknown function> + 0x5a02023 (0x7f8274dad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6372640Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8274db11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6373229Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8274da81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6373606Z frame #7: <unknown function> + 0x79d655 (0x7f8286298655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6373834Z frame #8: <unknown function> + 0xdc253 (0x7f823d2e3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6374041Z frame #9: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6374257Z frame #10: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6374265Z 
2025-07-17T11:07:25.6374360Z SIGABRT(6), PID: 1777266, Thread 1781543: 
2025-07-17T11:07:25.6374834Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6375048Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6375219Z frame #2: __poll + 0x4f (0x7fa32ef30bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6375435Z frame #3: <unknown function> + 0xd4540 (0x7fa1f3340540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6375640Z frame #4: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6375852Z frame #5: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6376005Z 
2025-07-17T11:07:25.6376106Z SIGABRT(6), PID: 1777266, Thread 1781551: 
2025-07-17T11:07:25.6376661Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6376878Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6377049Z frame #2: __poll + 0x4f (0x7fa32ef30bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6377338Z frame #3: <unknown function> + 0xd65a2 (0x7fa1f33425a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6377550Z frame #4: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6377758Z frame #5: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6377763Z 
2025-07-17T11:07:25.6377856Z SIGABRT(6), PID: 1777266, Thread 1781562: 
2025-07-17T11:07:25.6378321Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6378535Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6378740Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6378945Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6379155Z frame #4: <unknown function> + 0xd1e7f (0x7fa1f333de7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6379365Z frame #5: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6379570Z frame #6: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6379574Z 
2025-07-17T11:07:25.6379668Z SIGABRT(6), PID: 1777266, Thread 1781565: 
2025-07-17T11:07:25.6380125Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6380418Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6380622Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6380831Z frame #3: <unknown function> + 0x9ca53 (0x7fa32eeb4a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6381086Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55b7468d7bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6381316Z frame #5: <unknown function> + 0x1d314f (0x55b74699614f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6381534Z frame #6: <unknown function> + 0x1d2f6b (0x55b746995f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6381754Z frame #7: <unknown function> + 0x6df36 (0x55b746830f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6381996Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6382225Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6382466Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6382698Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6382930Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6383156Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6383390Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6383618Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6383932Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6384222Z frame #17: <unknown function> + 0x147734 (0x55b74690a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6384446Z frame #18: <unknown function> + 0x21ca82 (0x55b7469dfa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6384670Z frame #19: <unknown function> + 0x21c9e4 (0x55b7469df9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6384881Z frame #20: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6385164Z frame #21: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6385168Z 
2025-07-17T11:07:25.6385261Z SIGABRT(6), PID: 1777265, Thread 1781590: 
2025-07-17T11:07:25.6385740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6385953Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6386165Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6386366Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6386745Z frame #4: <unknown function> + 0x5a02023 (0x7f8af009f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6387315Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8af00a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6387908Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8af009a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6388288Z frame #7: <unknown function> + 0x79d655 (0x7f8b0158a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6388593Z frame #8: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6388801Z frame #9: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6389018Z frame #10: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6389024Z 
2025-07-17T11:07:25.6389113Z SIGABRT(6), PID: 1777266, Thread 1781576: 
2025-07-17T11:07:25.6389589Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6389799Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6389968Z frame #2: read + 0x4c (0x7fa32ef2c81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6390150Z frame #3: _Py_read + 0x4c (0x55b74699da6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6390382Z frame #4: <unknown function> + 0x208c3b (0x55b7469cbc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6390599Z frame #5: <unknown function> + 0x14648d (0x55b74690948d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6390845Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55b746918215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6391063Z frame #7: <unknown function> + 0x208ae7 (0x55b7469cbae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6391280Z frame #8: <unknown function> + 0x208472 (0x55b7469cb472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6391497Z frame #9: <unknown function> + 0x1da6ee (0x55b74699d6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6391723Z frame #10: <unknown function> + 0x1483bc (0x55b74690b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6392028Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6392322Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6392562Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55b7468ef3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6392782Z frame #14: <unknown function> + 0x147734 (0x55b74690a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6393027Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b7468f1c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6393314Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6393544Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6393769Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55b7468ff0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6394000Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55b7468ef7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6394225Z frame #20: <unknown function> + 0x147734 (0x55b74690a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6394443Z frame #21: <unknown function> + 0x21ca82 (0x55b7469dfa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6394662Z frame #22: <unknown function> + 0x21c9e4 (0x55b7469df9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6394874Z frame #23: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6395087Z frame #24: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6395090Z 
2025-07-17T11:07:25.6395181Z SIGABRT(6), PID: 1777272, Thread 1781595: 
2025-07-17T11:07:25.6395652Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6395866Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6396163Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6396368Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6396739Z frame #4: <unknown function> + 0x5a02023 (0x7f184e2f4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6397294Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f184e2f81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6397877Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f184e2ef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6398259Z frame #7: <unknown function> + 0x79d655 (0x7f185f7df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6398488Z frame #8: <unknown function> + 0xdc253 (0x7f181682a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6398696Z frame #9: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6398904Z frame #10: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6398909Z 
2025-07-17T11:07:25.6399003Z SIGABRT(6), PID: 1777271, Thread 1781633: 
2025-07-17T11:07:25.6399465Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6399675Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6399877Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6400144Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6400575Z frame #4: <unknown function> + 0x5a02023 (0x7f8274dad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6401130Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8274db11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6401777Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8274da81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6402153Z frame #7: <unknown function> + 0x79d655 (0x7f8286298655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6402381Z frame #8: <unknown function> + 0xdc253 (0x7f823d2e3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6402588Z frame #9: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6402800Z frame #10: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6402804Z 
2025-07-17T11:07:25.6402895Z SIGABRT(6), PID: 1777265, Thread 1781591: 
2025-07-17T11:07:25.6403360Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6403572Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6403775Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6403976Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6404342Z frame #4: <unknown function> + 0x5a02023 (0x7f8af009f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6404987Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8af00a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6405571Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8af009a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6405943Z frame #7: <unknown function> + 0x79d655 (0x7f8b0158a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6406169Z frame #8: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6406378Z frame #9: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6406594Z frame #10: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6406600Z 
2025-07-17T11:07:25.6406689Z SIGABRT(6), PID: 1777266, Thread 1781603: 
2025-07-17T11:07:25.6407159Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6407365Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6407570Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6407771Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6408136Z frame #4: <unknown function> + 0x5a02023 (0x7fa2732d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6408873Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa2732da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6409532Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa2732d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6409916Z frame #7: <unknown function> + 0x79d655 (0x7fa2847c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6410215Z frame #8: <unknown function> + 0xdc253 (0x7fa23b80c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6410426Z frame #9: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6410644Z frame #10: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6410648Z 
2025-07-17T11:07:25.6410741Z SIGABRT(6), PID: 1777272, Thread 1781596: 
2025-07-17T11:07:25.6411225Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6411434Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6411639Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6411836Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6412205Z frame #4: <unknown function> + 0x5a02023 (0x7f184e2f4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6412754Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f184e2f81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6413336Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f184e2ef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6413782Z frame #7: <unknown function> + 0x79d655 (0x7f185f7df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6414011Z frame #8: <unknown function> + 0xdc253 (0x7f181682a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6414216Z frame #9: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6414431Z frame #10: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6414435Z 
2025-07-17T11:07:25.6414528Z SIGABRT(6), PID: 1777271, Thread 1781636: 
2025-07-17T11:07:25.6414992Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6415201Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6415404Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6415602Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6416030Z frame #4: <unknown function> + 0x5a02023 (0x7f8274dad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6416584Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8274db11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6417165Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8274da81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6417627Z frame #7: <unknown function> + 0x79d655 (0x7f8286298655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6417930Z frame #8: <unknown function> + 0xdc253 (0x7f823d2e3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6418142Z frame #9: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6418354Z frame #10: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6418358Z 
2025-07-17T11:07:25.6418523Z SIGABRT(6), PID: 1777265, Thread 1781592: 
2025-07-17T11:07:25.6418992Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6419200Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6419402Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6419605Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6419969Z frame #4: <unknown function> + 0x5a02023 (0x7f8af009f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6420522Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8af00a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6421102Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8af009a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6421478Z frame #7: <unknown function> + 0x79d655 (0x7f8b0158a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6421704Z frame #8: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6421912Z frame #9: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6422206Z frame #10: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6422210Z 
2025-07-17T11:07:25.6422304Z SIGABRT(6), PID: 1777272, Thread 1781597: 
2025-07-17T11:07:25.6422766Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6422976Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6423175Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6423375Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6423742Z frame #4: <unknown function> + 0x5a02023 (0x7f184e2f4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6424290Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f184e2f81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6424872Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f184e2ef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6425247Z frame #7: <unknown function> + 0x79d655 (0x7f185f7df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6425475Z frame #8: <unknown function> + 0xdc253 (0x7f181682a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6425679Z frame #9: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6425958Z frame #10: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6425964Z 
2025-07-17T11:07:25.6426117Z SIGABRT(6), PID: 1777271, Thread 1781638: 
2025-07-17T11:07:25.6426589Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f823f076b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6426794Z frame #1: <unknown function> + 0x42520 (0x7f833092e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6426997Z frame #2: <unknown function> + 0x91117 (0x7f833097d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6427285Z frame #3: pthread_cond_wait + 0x211 (0x7f833097fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6427652Z frame #4: <unknown function> + 0x5a02023 (0x7f8274dad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6428202Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8274db11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6428787Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8274da81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6429156Z frame #7: <unknown function> + 0x79d655 (0x7f8286298655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6429386Z frame #8: <unknown function> + 0xdc253 (0x7f823d2e3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6429594Z frame #9: <unknown function> + 0x94ac3 (0x7f8330980ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6429809Z frame #10: <unknown function> + 0x126850 (0x7f8330a12850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6429813Z 
2025-07-17T11:07:25.6429904Z SIGABRT(6), PID: 1777266, Thread 1781604: 
2025-07-17T11:07:25.6430389Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6430670Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6430877Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6431077Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6431447Z frame #4: <unknown function> + 0x5a02023 (0x7fa2732d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6432000Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa2732da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6432583Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa2732d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6432963Z frame #7: <unknown function> + 0x79d655 (0x7fa2847c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6433189Z frame #8: <unknown function> + 0xdc253 (0x7fa23b80c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6433398Z frame #9: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6433612Z frame #10: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6433616Z 
2025-07-17T11:07:25.6433711Z SIGABRT(6), PID: 1777265, Thread 1781593: 
2025-07-17T11:07:25.6434178Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6434453Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6434722Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6434931Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6435298Z frame #4: <unknown function> + 0x5a02023 (0x7f8af009f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6435861Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8af00a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6436524Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8af009a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6436913Z frame #7: <unknown function> + 0x79d655 (0x7f8b0158a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6437151Z frame #8: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6437370Z frame #9: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6437587Z frame #10: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6437591Z 
2025-07-17T11:07:25.6437688Z SIGABRT(6), PID: 1777272, Thread 1781598: 
2025-07-17T11:07:25.6438163Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6438376Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6438578Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6438785Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6439233Z frame #4: <unknown function> + 0x5a02023 (0x7f184e2f4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6439798Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f184e2f81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6440392Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f184e2ef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6440775Z frame #7: <unknown function> + 0x79d655 (0x7f185f7df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6441009Z frame #8: <unknown function> + 0xdc253 (0x7f181682a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6441217Z frame #9: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6441431Z frame #10: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6441438Z 
2025-07-17T11:07:25.6441533Z SIGABRT(6), PID: 1777266, Thread 1781605: 
2025-07-17T11:07:25.6442009Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6442218Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6442421Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6442621Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6443061Z frame #4: <unknown function> + 0x5a02023 (0x7fa2732d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6443685Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa2732da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6444270Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa2732d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6444707Z frame #7: <unknown function> + 0x79d655 (0x7fa2847c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6444937Z frame #8: <unknown function> + 0xdc253 (0x7fa23b80c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6445144Z frame #9: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6445359Z frame #10: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6445367Z 
2025-07-17T11:07:25.6445460Z SIGABRT(6), PID: 1777265, Thread 1781594: 
2025-07-17T11:07:25.6445934Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8aba368b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6446146Z frame #1: <unknown function> + 0x42520 (0x7f8babc24520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6446358Z frame #2: <unknown function> + 0x91117 (0x7f8babc73117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6446561Z frame #3: pthread_cond_wait + 0x211 (0x7f8babc75a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6446937Z frame #4: <unknown function> + 0x5a02023 (0x7f8af009f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6447499Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8af00a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6448164Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8af009a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6448540Z frame #7: <unknown function> + 0x79d655 (0x7f8b0158a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6448777Z frame #8: <unknown function> + 0xdc253 (0x7f8ab85d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6448984Z frame #9: <unknown function> + 0x94ac3 (0x7f8babc76ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6449202Z frame #10: <unknown function> + 0x126850 (0x7f8babd08850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6449207Z 
2025-07-17T11:07:25.6449297Z SIGABRT(6), PID: 1777272, Thread 1781599: 
2025-07-17T11:07:25.6449771Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6449981Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6450185Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6450383Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6450757Z frame #4: <unknown function> + 0x5a02023 (0x7f184e2f4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6451311Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f184e2f81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6451959Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f184e2ef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6452405Z frame #7: <unknown function> + 0x79d655 (0x7f185f7df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6452636Z frame #8: <unknown function> + 0xdc253 (0x7f181682a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6452853Z frame #9: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6453127Z frame #10: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6453131Z 
2025-07-17T11:07:25.6453227Z SIGABRT(6), PID: 1777266, Thread 1781606: 
2025-07-17T11:07:25.6453699Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6453913Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6454119Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6454322Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6454690Z frame #4: <unknown function> + 0x5a02023 (0x7fa2732d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6455245Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa2732da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6455825Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa2732d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6456267Z frame #7: <unknown function> + 0x79d655 (0x7fa2847c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6456582Z frame #8: <unknown function> + 0xdc253 (0x7fa23b80c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6456798Z frame #9: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6457011Z frame #10: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6457015Z 
2025-07-17T11:07:25.6457111Z SIGABRT(6), PID: 1777272, Thread 1781600: 
2025-07-17T11:07:25.6457582Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6457792Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6457995Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6458198Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6458570Z frame #4: <unknown function> + 0x5a02023 (0x7f184e2f4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6459132Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f184e2f81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6459715Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f184e2ef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6460093Z frame #7: <unknown function> + 0x79d655 (0x7f185f7df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6460405Z frame #8: <unknown function> + 0xdc253 (0x7f181682a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6460620Z frame #9: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6460933Z frame #10: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6460938Z 
2025-07-17T11:07:25.6461034Z SIGABRT(6), PID: 1777266, Thread 1781607: 
2025-07-17T11:07:25.6461510Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6461793Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6462005Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6462206Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6462575Z frame #4: <unknown function> + 0x5a02023 (0x7fa2732d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6463138Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa2732da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6463727Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa2732d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6464104Z frame #7: <unknown function> + 0x79d655 (0x7fa2847c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6464336Z frame #8: <unknown function> + 0xdc253 (0x7fa23b80c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6464543Z frame #9: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6464764Z frame #10: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6464834Z 
2025-07-17T11:07:25.6464927Z SIGABRT(6), PID: 1777272, Thread 1781601: 
2025-07-17T11:07:25.6465405Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6465611Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6465815Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6466012Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6466382Z frame #4: <unknown function> + 0x5a02023 (0x7f184e2f4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6466938Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f184e2f81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6467530Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f184e2ef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6467905Z frame #7: <unknown function> + 0x79d655 (0x7f185f7df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6468136Z frame #8: <unknown function> + 0xdc253 (0x7f181682a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6468342Z frame #9: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6468561Z frame #10: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6468564Z 
2025-07-17T11:07:25.6468652Z SIGABRT(6), PID: 1777266, Thread 1781608: 
2025-07-17T11:07:25.6469194Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6469474Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6469680Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6469882Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6470309Z frame #4: <unknown function> + 0x5a02023 (0x7fa2732d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6470864Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa2732da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6471450Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa2732d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6471838Z frame #7: <unknown function> + 0x79d655 (0x7fa2847c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6472062Z frame #8: <unknown function> + 0xdc253 (0x7fa23b80c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6472279Z frame #9: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6472492Z frame #10: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6472496Z 
2025-07-17T11:07:25.6472588Z SIGABRT(6), PID: 1777266, Thread 1781609: 
2025-07-17T11:07:25.6473057Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6473270Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6473544Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6473748Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6474123Z frame #4: <unknown function> + 0x5a02023 (0x7fa2732d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6474692Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa2732da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6475275Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa2732d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6475658Z frame #7: <unknown function> + 0x79d655 (0x7fa2847c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6475884Z frame #8: <unknown function> + 0xdc253 (0x7fa23b80c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6476098Z frame #9: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6476309Z frame #10: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6476314Z 
2025-07-17T11:07:25.6476411Z SIGABRT(6), PID: 1777272, Thread 1781602: 
2025-07-17T11:07:25.6476882Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f18185bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6477090Z frame #1: <unknown function> + 0x42520 (0x7f1909e75520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6477292Z frame #2: <unknown function> + 0x91117 (0x7f1909ec4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6477565Z frame #3: pthread_cond_wait + 0x211 (0x7f1909ec6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6477995Z frame #4: <unknown function> + 0x5a02023 (0x7f184e2f4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6478560Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f184e2f81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6479211Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f184e2ef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6479585Z frame #7: <unknown function> + 0x79d655 (0x7f185f7df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6479818Z frame #8: <unknown function> + 0xdc253 (0x7f181682a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6480031Z frame #9: <unknown function> + 0x94ac3 (0x7f1909ec7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6480249Z frame #10: <unknown function> + 0x126850 (0x7f1909f59850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6480253Z 
2025-07-17T11:07:25.6480343Z SIGABRT(6), PID: 1777266, Thread 1781610: 
2025-07-17T11:07:25.6480814Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa23d59fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6481026Z frame #1: <unknown function> + 0x42520 (0x7fa32ee5a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6481231Z frame #2: <unknown function> + 0x91117 (0x7fa32eea9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6481429Z frame #3: pthread_cond_wait + 0x211 (0x7fa32eeaba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6481802Z frame #4: <unknown function> + 0x5a02023 (0x7fa2732d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6482426Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa2732da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6483011Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa2732d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6483388Z frame #7: <unknown function> + 0x79d655 (0x7fa2847c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6483616Z frame #8: <unknown function> + 0xdc253 (0x7fa23b80c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6483824Z frame #9: <unknown function> + 0x94ac3 (0x7fa32eeacac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6484041Z frame #10: <unknown function> + 0x126850 (0x7fa32ef3e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6484047Z 
2025-07-17T11:07:25.6484150Z ('RERUN', {'yellow': True}) [27.8516s] [ 39%]
2025-07-17T11:07:25.6484809Z distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph I0717 11:00:09.255000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1789276
2025-07-17T11:07:25.6485166Z I0717 11:00:09.256000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1789277
2025-07-17T11:07:25.6485515Z I0717 11:00:09.257000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1789278
2025-07-17T11:07:25.6485856Z I0717 11:00:09.306000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1789279
2025-07-17T11:07:25.6486264Z I0717 11:00:09.307000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1789280
2025-07-17T11:07:25.6486670Z I0717 11:00:09.308000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1789281
2025-07-17T11:07:25.6487022Z I0717 11:00:09.309000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1789282
2025-07-17T11:07:25.6487364Z I0717 11:00:09.310000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1789283
2025-07-17T11:07:25.6488386Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.6488559Z   import pkg_resources
2025-07-17T11:07:25.6489536Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.6489623Z   import pkg_resources
2025-07-17T11:07:25.6490588Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.6490671Z   import pkg_resources
2025-07-17T11:07:25.6491633Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.6491715Z   import pkg_resources
2025-07-17T11:07:25.6492755Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.6492838Z   import pkg_resources
2025-07-17T11:07:25.6493797Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.6493879Z   import pkg_resources
2025-07-17T11:07:25.6494849Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.6494933Z   import pkg_resources
2025-07-17T11:07:25.6495951Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.6496032Z   import pkg_resources
2025-07-17T11:07:25.6496515Z [rank4]:W0717 11:00:34.004000 1789280 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.6497085Z [rank7]:W0717 11:00:34.004000 1789283 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.6497631Z [rank1]:W0717 11:00:34.004000 1789277 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.6498098Z [rank2]:W0717 11:00:34.005000 1789278 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.6498564Z [rank5]:W0717 11:00:34.006000 1789281 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.6499100Z [rank6]:W0717 11:00:34.006000 1789282 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.6499570Z [rank3]:W0717 11:00:34.006000 1789279 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.6500034Z [rank0]:W0717 11:00:34.007000 1789276 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.6500328Z Memory access fault by GPU node-9 (Agent handle: 0x55af3a6e4bf0) on address 0x7f7fba804000. Reason: Unknown.
2025-07-17T11:07:25.6500601Z Memory access fault by GPU node-2 (Agent handle: 0x55b9ecaba2c0) on address 0x7fc962a1d000. Reason: Unknown.
2025-07-17T11:07:25.6500875Z Memory access fault by GPU node-8 (Agent handle: 0x5581f824ccb0) on address 0x7feee3e1d000. Reason: Unknown.
2025-07-17T11:07:25.6501138Z Memory access fault by GPU node-3 (Agent handle: 0x55f253c98460) on address 0x7f9407403000. Reason: Unknown.
2025-07-17T11:07:25.6501409Z Memory access fault by GPU node-6 (Agent handle: 0x55ab3c5ed5c0) on address 0x7f1a51c1e000. Reason: Unknown.
2025-07-17T11:07:25.6501673Z Memory access fault by GPU node-4 (Agent handle: 0x563745b8ea20) on address 0x7fbeb6a03000. Reason: Unknown.
2025-07-17T11:07:25.6501856Z SIGABRT(6), PID: 1789282, Thread 1789282: 
2025-07-17T11:07:25.6502346Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6502569Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6502807Z frame #2: <unknown function> + 0x773e7 (0x7ff728e983e7 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6503042Z frame #3: <unknown function> + 0x76c8e (0x7ff728e97c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6503267Z frame #4: <unknown function> + 0x6ad69 (0x7ff728e8bd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6503491Z frame #5: <unknown function> + 0x3f54db (0x7ff7b9af04db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6503703Z frame #6: <unknown function> + 0x3e11d8 (0x7ff7b9adc1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6503921Z frame #7: <unknown function> + 0xb2f86 (0x7ff7b97adf86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6504130Z frame #8: <unknown function> + 0x8e501 (0x7ff7b9789501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6504517Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7ff7e45907ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.6504905Z frame #10: <unknown function> + 0xd307d9 (0x7ff802ae37d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6505137Z frame #11: <unknown function> + 0x139902 (0x5581ed48b902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6505393Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5581ed482a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6505629Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6505941Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5581ed482a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6506234Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6506476Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6506704Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6506940Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6507236Z frame #19: <unknown function> + 0x1474b2 (0x5581ed4994b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6507440Z frame #20: PyObject_Call + 0xbc (0x5581ed499e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6507670Z frame #21: <unknown function> + 0x1ff69d (0x5581ed55169d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6507898Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5581ed487023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6508151Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5581ed482cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6508374Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6508615Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5581ed47e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6508834Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6509075Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5581ed47e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6509295Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6509537Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5581ed482a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6509792Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5581ed4864b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6510086Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5581ed497ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6510306Z frame #32: <unknown function> + 0x205949 (0x5581ed557949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6510529Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5581ed487023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6510769Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5581ed482cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6510996Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6511231Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5581ed47e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6511456Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6511692Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5581ed47e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6511920Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6512152Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5581ed47e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6512378Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6512620Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5581ed47f3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6512850Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6513241Z frame #44: <unknown function> + 0x7ab3f7 (0x7ff80255e3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6513465Z frame #45: <unknown function> + 0x13bc88 (0x5581ed48dc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6513736Z frame #46: PyObject_Call + 0x20f (0x5581ed499f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6514046Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5581ed48398d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6514268Z frame #48: <unknown function> + 0x147608 (0x5581ed499608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6514516Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5581ed480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6514738Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6515041Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5581ed47e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6515263Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6515506Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5581ed47f3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6515732Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6515976Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5581ed47e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6516198Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6516439Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5581ed480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6516664Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6516909Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5581ed480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6517130Z frame #60: <unknown function> + 0x147608 (0x5581ed499608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6517370Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5581ed480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6517586Z frame #62: <unknown function> + 0x147608 (0x5581ed499608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6517658Z 
2025-07-17T11:07:25.6517764Z SIGABRT(6), PID: 1789282, Thread 1789599: 
2025-07-17T11:07:25.6518244Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6518464Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6518670Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6518877Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6519388Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6519604Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6519816Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6519828Z 
2025-07-17T11:07:25.6519922Z SIGABRT(6), PID: 1789282, Thread 1789600: 
2025-07-17T11:07:25.6520396Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6520606Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6520816Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6521018Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6521520Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6521795Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6522081Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6522086Z 
2025-07-17T11:07:25.6522179Z SIGABRT(6), PID: 1789282, Thread 1789601: 
2025-07-17T11:07:25.6522654Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6522928Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6523141Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6523341Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6523842Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6524052Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6524268Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6524272Z 
2025-07-17T11:07:25.6524360Z SIGABRT(6), PID: 1789282, Thread 1789603: 
2025-07-17T11:07:25.6524830Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6525038Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6525248Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6525446Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6525941Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6526244Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6526463Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6526467Z 
2025-07-17T11:07:25.6526557Z SIGABRT(6), PID: 1789282, Thread 1789605: 
2025-07-17T11:07:25.6527034Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6527243Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6527452Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6527656Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6528156Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6528365Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6528584Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6528589Z 
2025-07-17T11:07:25.6528681Z SIGABRT(6), PID: 1789282, Thread 1789606: 
2025-07-17T11:07:25.6529153Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6529361Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6529571Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6529837Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6530397Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6530618Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6530828Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6530892Z 
2025-07-17T11:07:25.6530991Z SIGABRT(6), PID: 1789282, Thread 1789608: 
2025-07-17T11:07:25.6531458Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6531670Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6531878Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6532081Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6532574Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6532783Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6532990Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6532995Z 
2025-07-17T11:07:25.6533091Z SIGABRT(6), PID: 1789282, Thread 1789610: 
2025-07-17T11:07:25.6533553Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6533764Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6533973Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6534245Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6534739Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6534952Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6535165Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6535169Z 
2025-07-17T11:07:25.6535263Z SIGABRT(6), PID: 1789282, Thread 1789612: 
2025-07-17T11:07:25.6535731Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6536079Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6536291Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6536493Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6536992Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6537206Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6537417Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6537422Z 
2025-07-17T11:07:25.6537518Z SIGABRT(6), PID: 1789282, Thread 1789614: 
2025-07-17T11:07:25.6538070Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6538286Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6538567Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6538776Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6539266Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6539553Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6539764Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6539768Z 
2025-07-17T11:07:25.6539859Z SIGABRT(6), PID: 1789282, Thread 1789616: 
2025-07-17T11:07:25.6540332Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6540544Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6540748Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6540953Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6541446Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6541657Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6541871Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6541875Z 
2025-07-17T11:07:25.6541964Z SIGABRT(6), PID: 1789282, Thread 1789618: 
2025-07-17T11:07:25.6542438Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6542846Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6543058Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6543256Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6543751Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6543959Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6544172Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6544176Z 
2025-07-17T11:07:25.6544264Z SIGABRT(6), PID: 1789282, Thread 1789620: 
2025-07-17T11:07:25.6544738Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6544945Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6545156Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6545359Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6545863Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6546072Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6546353Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6546358Z 
2025-07-17T11:07:25.6546454Z SIGABRT(6), PID: 1789282, Thread 1789622: 
2025-07-17T11:07:25.6546992Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6547203Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6547413Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6547676Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6548175Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6548383Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6548600Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6548606Z 
2025-07-17T11:07:25.6548699Z SIGABRT(6), PID: 1789282, Thread 1789624: 
2025-07-17T11:07:25.6549171Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6549378Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6549589Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6549787Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6550288Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6550500Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6550791Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6550795Z 
2025-07-17T11:07:25.6550883Z SIGABRT(6), PID: 1789282, Thread 1789626: 
2025-07-17T11:07:25.6551353Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6551566Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6551770Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6551969Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6552462Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6552677Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6552890Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6552895Z 
2025-07-17T11:07:25.6552988Z SIGABRT(6), PID: 1789282, Thread 1789628: 
2025-07-17T11:07:25.6553452Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6553666Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6553870Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6554071Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6554632Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6554909Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6555120Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6555125Z 
2025-07-17T11:07:25.6555221Z SIGABRT(6), PID: 1789282, Thread 1789630: 
2025-07-17T11:07:25.6555685Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6555985Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6556191Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6556392Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6556885Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6557102Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6557314Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6557318Z 
2025-07-17T11:07:25.6557410Z SIGABRT(6), PID: 1789282, Thread 1789632: 
2025-07-17T11:07:25.6557878Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6558094Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6558298Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6558503Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6559073Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6559288Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6559498Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6559504Z 
2025-07-17T11:07:25.6559600Z SIGABRT(6), PID: 1789282, Thread 1789634: 
2025-07-17T11:07:25.6560067Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6560277Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6560485Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6560688Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6561186Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6561399Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6561609Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6561615Z 
2025-07-17T11:07:25.6561709Z SIGABRT(6), PID: 1789282, Thread 1789636: 
2025-07-17T11:07:25.6562182Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6562388Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6562664Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6562929Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6563431Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6563640Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6563915Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6563919Z 
2025-07-17T11:07:25.6564009Z SIGABRT(6), PID: 1789282, Thread 1789638: 
2025-07-17T11:07:25.6564483Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6564693Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6564910Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6565108Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6565602Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6565813Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6566029Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6566035Z 
2025-07-17T11:07:25.6566124Z SIGABRT(6), PID: 1789282, Thread 1789640: 
2025-07-17T11:07:25.6566597Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6566805Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6567088Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6567283Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6567779Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6567989Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6568205Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6568209Z 
2025-07-17T11:07:25.6568298Z SIGABRT(6), PID: 1789282, Thread 1789642: 
2025-07-17T11:07:25.6568770Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6568980Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6569189Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6569389Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6569886Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6570104Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6570319Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6570323Z 
2025-07-17T11:07:25.6570412Z SIGABRT(6), PID: 1789282, Thread 1789644: 
2025-07-17T11:07:25.6571031Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6571245Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6571453Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6571652Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6572219Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6572426Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6572641Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6572645Z 
2025-07-17T11:07:25.6572738Z SIGABRT(6), PID: 1789282, Thread 1789646: 
2025-07-17T11:07:25.6573211Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6573425Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6573630Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6573835Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6574330Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6574546Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6574759Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6574763Z 
2025-07-17T11:07:25.6574931Z SIGABRT(6), PID: 1789282, Thread 1789647: 
2025-07-17T11:07:25.6575404Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6575617Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6575825Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6576082Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6576578Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6576792Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6577007Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6577013Z 
2025-07-17T11:07:25.6577299Z Memory access fault by GPU node-5 (Agent handle: 0x564255ad9980) on address 0x7fa027e05000. Reason: Unknown.
2025-07-17T11:07:25.6577391Z SIGABRT(6), PID: 1789282, Thread 1789649: 
2025-07-17T11:07:25.6577866Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6578079Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6578289Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6578487Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6579071Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6579360Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6579578Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6579582Z 
2025-07-17T11:07:25.6579675Z SIGABRT(6), PID: 1789282, Thread 1789651: 
2025-07-17T11:07:25.6580150Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6580432Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6580641Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6580840Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6581342Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6581553Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6581773Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6581776Z 
2025-07-17T11:07:25.6581866Z SIGABRT(6), PID: 1789282, Thread 1789653: 
2025-07-17T11:07:25.6582338Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6582546Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6582756Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6582955Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6583454Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6583747Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6583962Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6583965Z 
2025-07-17T11:07:25.6584055Z SIGABRT(6), PID: 1789282, Thread 1789655: 
2025-07-17T11:07:25.6584527Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6584738Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6584943Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6585147Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6585645Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6585857Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6586065Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6586070Z 
2025-07-17T11:07:25.6586166Z SIGABRT(6), PID: 1789282, Thread 1789657: 
2025-07-17T11:07:25.6586630Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6586843Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6587113Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6587402Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6587919Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6588132Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6588419Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6588423Z 
2025-07-17T11:07:25.6588517Z SIGABRT(6), PID: 1789282, Thread 1789659: 
2025-07-17T11:07:25.6588984Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6589196Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6589402Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6589606Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6590101Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6590318Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6590529Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6590533Z 
2025-07-17T11:07:25.6590624Z SIGABRT(6), PID: 1789282, Thread 1789661: 
2025-07-17T11:07:25.6591096Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6591308Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6591585Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6591791Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6592285Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6592503Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6592714Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6592718Z 
2025-07-17T11:07:25.6592812Z SIGABRT(6), PID: 1789282, Thread 1789663: 
2025-07-17T11:07:25.6593279Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6593497Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6593704Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6593909Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6594402Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6594618Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6594828Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6594837Z 
2025-07-17T11:07:25.6594923Z SIGABRT(6), PID: 1789282, Thread 1789665: 
2025-07-17T11:07:25.6595464Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6595732Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6595947Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6596147Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6596638Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6596909Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6597125Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6597129Z 
2025-07-17T11:07:25.6597222Z SIGABRT(6), PID: 1789282, Thread 1789667: 
2025-07-17T11:07:25.6597700Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6597910Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6598115Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6598312Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6598809Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6599017Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6599231Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6599238Z 
2025-07-17T11:07:25.6599331Z SIGABRT(6), PID: 1789282, Thread 1789669: 
2025-07-17T11:07:25.6599873Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6600081Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6600291Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6600491Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6600988Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6601195Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6601408Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6601414Z 
2025-07-17T11:07:25.6601505Z SIGABRT(6), PID: 1789282, Thread 1789671: 
2025-07-17T11:07:25.6601977Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6602184Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6602392Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6602592Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6603090Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6603366Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6603586Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6603649Z 
2025-07-17T11:07:25.6603741Z SIGABRT(6), PID: 1789282, Thread 1789672: 
2025-07-17T11:07:25.6604213Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6604419Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6604695Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6604894Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6605391Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6605604Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6605822Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6605826Z 
2025-07-17T11:07:25.6605918Z SIGABRT(6), PID: 1789282, Thread 1789674: 
2025-07-17T11:07:25.6606379Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6606593Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6606797Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6606999Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6607493Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6607782Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6607992Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6607996Z 
2025-07-17T11:07:25.6608086Z SIGABRT(6), PID: 1789282, Thread 1789676: 
2025-07-17T11:07:25.6608554Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6608762Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6608964Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6609164Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6609657Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6609874Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6610085Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6610088Z 
2025-07-17T11:07:25.6610185Z SIGABRT(6), PID: 1789282, Thread 1789678: 
2025-07-17T11:07:25.6610648Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6610868Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6611071Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6611273Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6611887Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6612106Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6612320Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6612324Z 
2025-07-17T11:07:25.6612418Z SIGABRT(6), PID: 1789282, Thread 1789680: 
2025-07-17T11:07:25.6612947Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6613159Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6613362Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6613570Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6614061Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6614274Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6614484Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6614489Z 
2025-07-17T11:07:25.6614587Z SIGABRT(6), PID: 1789282, Thread 1789682: 
2025-07-17T11:07:25.6615052Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6615262Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6615465Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6615666Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6616286Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6616499Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6616708Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6616717Z 
2025-07-17T11:07:25.6616805Z SIGABRT(6), PID: 1789282, Thread 1789684: 
2025-07-17T11:07:25.6617274Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6617481Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6617687Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6617893Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6618390Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6618600Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6618816Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6618820Z 
2025-07-17T11:07:25.6618908Z SIGABRT(6), PID: 1789282, Thread 1789687: 
2025-07-17T11:07:25.6619441Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6619763Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6620055Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6620266Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6620770Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6621057Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6621275Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6621279Z 
2025-07-17T11:07:25.6621369Z SIGABRT(6), PID: 1789282, Thread 1789689: 
2025-07-17T11:07:25.6621846Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6622060Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6622272Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6622468Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6622963Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6623173Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6623384Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6623388Z 
2025-07-17T11:07:25.6623477Z SIGABRT(6), PID: 1789282, Thread 1789691: 
2025-07-17T11:07:25.6623949Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6624240Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6624449Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6624649Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6625145Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6625355Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6625571Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6625576Z 
2025-07-17T11:07:25.6625666Z SIGABRT(6), PID: 1789282, Thread 1789694: 
2025-07-17T11:07:25.6626137Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6626345Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6626555Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6626752Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6627249Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6627464Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6627674Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6627678Z 
2025-07-17T11:07:25.6627844Z SIGABRT(6), PID: 1789282, Thread 1789696: 
2025-07-17T11:07:25.6628374Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6628586Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6628794Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6629060Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6629557Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6629771Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6629983Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6629987Z 
2025-07-17T11:07:25.6630083Z SIGABRT(6), PID: 1789282, Thread 1789698: 
2025-07-17T11:07:25.6630555Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6630780Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6630988Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6631193Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6631684Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6631896Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6632104Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6632174Z 
2025-07-17T11:07:25.6632272Z SIGABRT(6), PID: 1789282, Thread 1789700: 
2025-07-17T11:07:25.6632741Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6632949Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6633158Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6633360Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6633850Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6634064Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6634278Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6634282Z 
2025-07-17T11:07:25.6634376Z SIGABRT(6), PID: 1789282, Thread 1789702: 
2025-07-17T11:07:25.6634842Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6635053Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6635259Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6635464Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6636030Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6636251Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6636520Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6636524Z 
2025-07-17T11:07:25.6636617Z SIGABRT(6), PID: 1789282, Thread 1789704: 
2025-07-17T11:07:25.6637082Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6637359Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6637566Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6637769Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6638269Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6638484Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6638699Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6638703Z 
2025-07-17T11:07:25.6638791Z SIGABRT(6), PID: 1789282, Thread 1789706: 
2025-07-17T11:07:25.6639258Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6639469Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6639677Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6639879Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6640379Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6640663Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6640877Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6640881Z 
2025-07-17T11:07:25.6640970Z SIGABRT(6), PID: 1789282, Thread 1789708: 
2025-07-17T11:07:25.6641440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6641650Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6641864Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6642065Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6642558Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6642768Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6642986Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6642992Z 
2025-07-17T11:07:25.6643083Z SIGABRT(6), PID: 1789282, Thread 1789710: 
2025-07-17T11:07:25.6643552Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6643758Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6644037Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6644238Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6644797Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6645006Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6645225Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6645289Z 
2025-07-17T11:07:25.6645382Z SIGABRT(6), PID: 1789282, Thread 1789712: 
2025-07-17T11:07:25.6645855Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6646062Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6646278Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6646480Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6646976Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6647186Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6647399Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6647403Z 
2025-07-17T11:07:25.6647494Z SIGABRT(6), PID: 1789282, Thread 1789714: 
2025-07-17T11:07:25.6647967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6648181Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6648476Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6648676Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6649169Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6649382Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6649594Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6649598Z 
2025-07-17T11:07:25.6649694Z SIGABRT(6), PID: 1789282, Thread 1789716: 
2025-07-17T11:07:25.6650161Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6650375Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6650584Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6650785Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6651276Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6651491Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6651700Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6651704Z 
2025-07-17T11:07:25.6651795Z SIGABRT(6), PID: 1789282, Thread 1789719: 
2025-07-17T11:07:25.6652357Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6652642Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6652853Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6653058Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6653550Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6653826Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6654037Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6654041Z 
2025-07-17T11:07:25.6654133Z SIGABRT(6), PID: 1789282, Thread 1789721: 
2025-07-17T11:07:25.6654602Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6654817Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6655022Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6655225Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6655718Z frame #4: <unknown function> + 0x3475ab (0x7ff8a9ad35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6655976Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6656184Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6656188Z 
2025-07-17T11:07:25.6656284Z SIGABRT(6), PID: 1789282, Thread 1789807: 
2025-07-17T11:07:25.6656754Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6657297Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7ff7bb32f08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6657505Z frame #2: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6657709Z frame #3: pthread_kill + 0x12c (0x7ff8acc3d9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6657874Z frame #4: raise + 0x16 (0x7ff8acbe9476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6658038Z frame #5: abort + 0xd3 (0x7ff8acbcf7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6658275Z frame #6: <unknown function> + 0x972cc (0x7ff728eb82cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6658510Z frame #7: <unknown function> + 0x95f47 (0x7ff728eb6f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6658737Z frame #8: <unknown function> + 0x34b71 (0x7ff728e55b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6658956Z frame #9: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6659175Z frame #10: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6659271Z SIGABRT(6), PID: 1789276, Thread 1789276: 
2025-07-17T11:07:25.6659740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6659951Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6660174Z frame #2: <unknown function> + 0x7743a (0x7fd19eb0943a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6660481Z frame #3: <unknown function> + 0x76c8e (0x7fd19eb08c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6660790Z frame #4: <unknown function> + 0x6ad69 (0x7fd19eafcd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6661020Z frame #5: <unknown function> + 0x3f54db (0x7fd22f7614db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6661238Z frame #6: <unknown function> + 0x3e11d8 (0x7fd22f74d1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6661462Z frame #7: <unknown function> + 0xb2f86 (0x7fd22f41ef86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6661748Z frame #8: <unknown function> + 0x8e501 (0x7fd22f3fa501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6662148Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fd25a2017ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.6662534Z frame #10: <unknown function> + 0xd307d9 (0x7fd2787547d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6662776Z frame #11: <unknown function> + 0x139902 (0x55b9e29a3902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6663032Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55b9e299aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6663267Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6663513Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55b9e299aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6663748Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6663991Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6664222Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6664458Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6664690Z frame #19: <unknown function> + 0x1474b2 (0x55b9e29b14b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6664969Z frame #20: PyObject_Call + 0xbc (0x55b9e29b1e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6665198Z frame #21: <unknown function> + 0x1ff69d (0x55b9e2a6969d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6665423Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55b9e299f023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6665669Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55b9e299acc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6665893Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6666132Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55b9e29963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6666354Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6666595Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55b9e29963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6666823Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6667071Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55b9e299aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6667327Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55b9e299e4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6667551Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55b9e29afab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6667776Z frame #32: <unknown function> + 0x205949 (0x55b9e2a6f949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6668003Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55b9e299f023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6668241Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55b9e299acc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6668534Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6668833Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55b9e29963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6669066Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6669300Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55b9e29963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6669527Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6669946Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55b9e29963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6670173Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6670413Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55b9e29973f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6670645Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6671044Z frame #44: <unknown function> + 0x7ab3f7 (0x7fd2781cf3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6671268Z frame #45: <unknown function> + 0x13bc88 (0x55b9e29a5c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6671476Z frame #46: PyObject_Call + 0x20f (0x55b9e29b1f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6671724Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55b9e299b98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6671943Z frame #48: <unknown function> + 0x147608 (0x55b9e29b1608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6672188Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b9e2998c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6672412Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6672653Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55b9e29963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6672945Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6673186Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55b9e29973f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6673408Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6673645Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55b9e29963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6673868Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6674110Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b9e2998c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6674332Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6674580Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b9e2998c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6674801Z frame #60: <unknown function> + 0x147608 (0x55b9e29b1608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6675044Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b9e2998c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6675260Z frame #62: <unknown function> + 0x147608 (0x55b9e29b1608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6675264Z 
2025-07-17T11:07:25.6675272Z 
2025-07-17T11:07:25.6675366Z SIGABRT(6), PID: 1789282, Thread 1792719: 
2025-07-17T11:07:25.6675851Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6676066Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6676242Z frame #2: __poll + 0x4f (0x7ff8accbfbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6676720Z frame #3: <unknown function> + 0x489f (0x7ff8ac7cb89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.6677015Z frame #4: <unknown function> + 0x1483bc (0x5581ed49a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6677260Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6677495Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6677810Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6678038Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6678281Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5581ed482a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6678513Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6678751Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5581ed480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6678982Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6679216Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6679444Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6679676Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6679898Z frame #16: <unknown function> + 0x147734 (0x5581ed499734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6680118Z frame #17: <unknown function> + 0x21ca82 (0x5581ed56ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6680340Z frame #18: <unknown function> + 0x21c9e4 (0x5581ed56e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6680557Z frame #19: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6680851Z frame #20: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6680855Z 
2025-07-17T11:07:25.6680948Z SIGABRT(6), PID: 1789276, Thread 1789284: 
2025-07-17T11:07:25.6681430Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6681643Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6681855Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6682056Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6682565Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6682777Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6682994Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6682999Z 
2025-07-17T11:07:25.6683086Z SIGABRT(6), PID: 1789276, Thread 1789285: 
2025-07-17T11:07:25.6683564Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6683771Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6683976Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6684175Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6684745Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6685022Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6685243Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6685247Z 
2025-07-17T11:07:25.6685336Z SIGABRT(6), PID: 1789276, Thread 1789286: 
2025-07-17T11:07:25.6685885Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6686095Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6686307Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6686512Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6687019Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6687228Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6687441Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6687445Z 
2025-07-17T11:07:25.6687538Z SIGABRT(6), PID: 1789282, Thread 1793523: 
2025-07-17T11:07:25.6688018Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6688224Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6688435Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6688660Z frame #3: pthread_cond_clockwait + 0x1fd (0x7ff8acc3b2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6689178Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7ff7e7077048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6689600Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7ff7e7079bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6689836Z frame #6: <unknown function> + 0xdc253 (0x7ff7b959b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6690046Z frame #7: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6690267Z frame #8: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6690271Z 
2025-07-17T11:07:25.6690361Z SIGABRT(6), PID: 1789283, Thread 1789283: 
2025-07-17T11:07:25.6690832Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6691046Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6691278Z frame #2: <unknown function> + 0x7741f (0x7f87f2c1541f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6691509Z frame #3: <unknown function> + 0x76c8e (0x7f87f2c14c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6691734Z frame #4: <unknown function> + 0x6ad69 (0x7f87f2c08d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6691949Z frame #5: <unknown function> + 0x3f54db (0x7f888386d4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6692158Z frame #6: <unknown function> + 0x3e11d8 (0x7f88838591d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6692371Z frame #7: <unknown function> + 0xb2f86 (0x7f888352af86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6692644Z frame #8: <unknown function> + 0x8e501 (0x7f8883506501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.6693096Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f88ae30d7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.6693486Z frame #10: <unknown function> + 0xd307d9 (0x7f88cc8607d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6693721Z frame #11: <unknown function> + 0x139902 (0x55af30a01902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6694029Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55af309f8a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6694260Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6694500Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55af309f8a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6694732Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6694968Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6695199Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6695434Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6695660Z frame #19: <unknown function> + 0x1474b2 (0x55af30a0f4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6695863Z frame #20: PyObject_Call + 0xbc (0x55af30a0fe4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6696207Z frame #21: <unknown function> + 0x1ff69d (0x55af30ac769d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6696433Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55af309fd023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6696677Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55af309f8cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6696901Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6697235Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55af309f43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6697454Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6697688Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55af309f43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6697910Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6698157Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55af309f8a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6698415Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55af309fc4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6698643Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55af30a0dab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6698865Z frame #32: <unknown function> + 0x205949 (0x55af30acd949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6699092Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55af309fd023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6699340Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55af309f8cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6699566Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6699807Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55af309f43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6700031Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6700267Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55af309f43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6700489Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6700808Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55af309f43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6701109Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6701358Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55af309f53f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6701583Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6701974Z frame #44: <unknown function> + 0x7ab3f7 (0x7f88cc2db3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6702277Z frame #45: <unknown function> + 0x13bc88 (0x55af30a03c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6702484Z frame #46: PyObject_Call + 0x20f (0x55af30a0ff9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6702724Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55af309f998d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6702952Z frame #48: <unknown function> + 0x147608 (0x55af30a0f608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6703195Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55af309f6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6703422Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6703657Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55af309f43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6703887Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6704128Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55af309f53f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6704359Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6704596Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55af309f43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6704823Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6705134Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55af309f6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6705363Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6705609Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55af309f6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6705838Z frame #60: <unknown function> + 0x147608 (0x55af30a0f608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6706081Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55af309f6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6706299Z frame #62: <unknown function> + 0x147608 (0x55af30a0f608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6706394Z SIGABRT(6), PID: 1789276, Thread 1789287: 
2025-07-17T11:07:25.6706878Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6707100Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6707309Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6707518Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6708021Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6708241Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6708458Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6708463Z 
2025-07-17T11:07:25.6708466Z 
2025-07-17T11:07:25.6708638Z SIGABRT(6), PID: 1789282, Thread 1793524: 
2025-07-17T11:07:25.6709197Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6709423Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6709627Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6709859Z frame #3: pthread_cond_clockwait + 0x1fd (0x7ff8acc3b2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6710399Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7ff7e7071741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.6710639Z frame #5: <unknown function> + 0xdc253 (0x7ff7b959b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6710850Z frame #6: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6711070Z frame #7: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6711166Z SIGABRT(6), PID: 1789276, Thread 1789288: 
2025-07-17T11:07:25.6711647Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6711855Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6712069Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6712272Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6712781Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6712990Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6713288Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6713383Z SIGABRT(6), PID: 1789283, Thread 1789602: 
2025-07-17T11:07:25.6713862Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6714072Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6714288Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6714484Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6714989Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6715196Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6715415Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6715420Z 
2025-07-17T11:07:25.6715423Z 
2025-07-17T11:07:25.6715426Z 
2025-07-17T11:07:25.6715518Z SIGABRT(6), PID: 1789282, Thread 1793525: 
2025-07-17T11:07:25.6715995Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6716208Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6716389Z frame #2: ioctl + 0x3f (0x7ff8accc194f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6716624Z frame #3: <unknown function> + 0x13eef0 (0x7ff728f5fef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6716926Z frame #4: <unknown function> + 0x1378c4 (0x7ff728f588c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6717173Z frame #5: <unknown function> + 0x95b2b (0x7ff728eb6b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6717472Z frame #6: <unknown function> + 0x34b71 (0x7ff728e55b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.6717689Z frame #7: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6717910Z frame #8: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6717973Z 
2025-07-17T11:07:25.6718069Z SIGABRT(6), PID: 1789283, Thread 1789604: 
2025-07-17T11:07:25.6718548Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6718759Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6718965Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6719174Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6719676Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6719882Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6720095Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6720100Z 
2025-07-17T11:07:25.6720197Z SIGABRT(6), PID: 1789276, Thread 1789289: 
2025-07-17T11:07:25.6720668Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6720879Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6721087Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6721365Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6721861Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6722076Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6722289Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6722293Z 
2025-07-17T11:07:25.6722395Z SIGABRT(6), PID: 1789282, Thread 1793554: 
2025-07-17T11:07:25.6722862Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6723079Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6723252Z frame #2: __poll + 0x4f (0x7ff8accbfbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6723474Z frame #3: <unknown function> + 0xd4540 (0x7ff7710cf540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6723689Z frame #4: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6723908Z frame #5: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6723916Z 
2025-07-17T11:07:25.6724005Z SIGABRT(6), PID: 1789283, Thread 1789607: 
2025-07-17T11:07:25.6724479Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6724684Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6724963Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6725240Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6725744Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6725952Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6726170Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6726234Z 
2025-07-17T11:07:25.6726326Z SIGABRT(6), PID: 1789276, Thread 1789290: 
2025-07-17T11:07:25.6726801Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6727012Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6727228Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6727430Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6727933Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6728145Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6728362Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6728366Z 
2025-07-17T11:07:25.6728458Z SIGABRT(6), PID: 1789282, Thread 1793561: 
2025-07-17T11:07:25.6728932Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6729142Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6729405Z frame #2: __poll + 0x4f (0x7ff8accbfbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6729625Z frame #3: <unknown function> + 0xd65a2 (0x7ff7710d15a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6729844Z frame #4: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6730059Z frame #5: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6730071Z 
2025-07-17T11:07:25.6730161Z SIGABRT(6), PID: 1789282, Thread 1793569: 
2025-07-17T11:07:25.6730687Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6736655Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6736897Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6737122Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6737362Z frame #4: <unknown function> + 0xd1e7f (0x7ff7710cce7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.6737582Z frame #5: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6737793Z frame #6: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6737804Z 
2025-07-17T11:07:25.6737905Z SIGABRT(6), PID: 1789283, Thread 1789609: 
2025-07-17T11:07:25.6738408Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6738625Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6738971Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6739267Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6739791Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6740002Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6740300Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6740400Z SIGABRT(6), PID: 1789276, Thread 1789291: 
2025-07-17T11:07:25.6740886Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6741092Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6741300Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6741504Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6742007Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6742213Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6742426Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6742431Z 
2025-07-17T11:07:25.6742523Z SIGABRT(6), PID: 1789282, Thread 1793578: 
2025-07-17T11:07:25.6742997Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6743221Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6743551Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6743758Z frame #3: <unknown function> + 0x9ca53 (0x7ff8acc43a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6744019Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5581ed466bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6744253Z frame #5: <unknown function> + 0x1d314f (0x5581ed52514f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6744478Z frame #6: <unknown function> + 0x1d2f6b (0x5581ed524f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6744698Z frame #7: <unknown function> + 0x6df36 (0x5581ed3bff36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6744943Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6745169Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6745418Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6745652Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6745888Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6746110Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6746344Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6746565Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6746799Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6747087Z frame #17: <unknown function> + 0x147734 (0x5581ed499734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6747315Z frame #18: <unknown function> + 0x21ca82 (0x5581ed56ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6747594Z frame #19: <unknown function> + 0x21c9e4 (0x5581ed56e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6747818Z frame #20: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6748033Z frame #21: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6748098Z 
2025-07-17T11:07:25.6748101Z 
2025-07-17T11:07:25.6748203Z SIGABRT(6), PID: 1789283, Thread 1789611: 
2025-07-17T11:07:25.6748690Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6748905Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6749113Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6749322Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6749825Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6750034Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6750243Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6750350Z SIGABRT(6), PID: 1789282, Thread 1793593: 
2025-07-17T11:07:25.6750838Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6751058Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6751236Z frame #2: read + 0x4c (0x7ff8accbb81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6751497Z frame #3: _Py_read + 0x4c (0x5581ed52ca6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6751730Z frame #4: <unknown function> + 0x208c3b (0x5581ed55ac3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6751953Z frame #5: <unknown function> + 0x14648d (0x5581ed49848d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6752199Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5581ed4a7215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6752424Z frame #7: <unknown function> + 0x208ae7 (0x5581ed55aae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6752639Z frame #8: <unknown function> + 0x208472 (0x5581ed55a472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6752858Z frame #9: <unknown function> + 0x1da6ee (0x5581ed52c6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6753084Z frame #10: <unknown function> + 0x1483bc (0x5581ed49a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6753334Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6753563Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6753802Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5581ed47e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6754020Z frame #14: <unknown function> + 0x147734 (0x5581ed499734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6754271Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5581ed480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6754492Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6754728Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6755026Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5581ed48e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6755263Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5581ed47e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6755544Z frame #20: <unknown function> + 0x147734 (0x5581ed499734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6755771Z frame #21: <unknown function> + 0x21ca82 (0x5581ed56ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6755990Z frame #22: <unknown function> + 0x21c9e4 (0x5581ed56e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.6756280Z frame #23: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6756497Z frame #24: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6756501Z 
2025-07-17T11:07:25.6756505Z 
2025-07-17T11:07:25.6756604Z SIGABRT(6), PID: 1789276, Thread 1789292: 
2025-07-17T11:07:25.6757093Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6757312Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6757522Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6757730Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6758252Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6758469Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6758678Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6758686Z 
2025-07-17T11:07:25.6758779Z SIGABRT(6), PID: 1789283, Thread 1789613: 
2025-07-17T11:07:25.6759257Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6759539Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6759746Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6759946Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6760451Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6760657Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6760866Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6760958Z SIGABRT(6), PID: 1789276, Thread 1789293: 
2025-07-17T11:07:25.6761433Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6761641Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6761846Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6762045Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6762547Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6762753Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6762965Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6763040Z 
2025-07-17T11:07:25.6763045Z 
2025-07-17T11:07:25.6763138Z SIGABRT(6), PID: 1789283, Thread 1789615: 
2025-07-17T11:07:25.6763666Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6763873Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6764078Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6764335Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6764833Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6765038Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6765247Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6765253Z 
2025-07-17T11:07:25.6765342Z SIGABRT(6), PID: 1789276, Thread 1789294: 
2025-07-17T11:07:25.6765811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6766018Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6766224Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6766419Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6766912Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6767120Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6767333Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6767402Z 
2025-07-17T11:07:25.6767493Z SIGABRT(6), PID: 1789283, Thread 1789617: 
2025-07-17T11:07:25.6767963Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6768166Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6768373Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6768567Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6769061Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6769267Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6769479Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6769484Z 
2025-07-17T11:07:25.6769572Z SIGABRT(6), PID: 1789276, Thread 1789295: 
2025-07-17T11:07:25.6770038Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6770249Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6770459Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6770656Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6771215Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6771492Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6771703Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6771707Z 
2025-07-17T11:07:25.6771798Z SIGABRT(6), PID: 1789283, Thread 1789619: 
2025-07-17T11:07:25.6772259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6772548Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6772751Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6772948Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6773443Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6773655Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6773861Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6773865Z 
2025-07-17T11:07:25.6773957Z SIGABRT(6), PID: 1789276, Thread 1789296: 
2025-07-17T11:07:25.6774423Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6774630Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6774831Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6775033Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6775526Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6775811Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6776080Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6776084Z 
2025-07-17T11:07:25.6776181Z SIGABRT(6), PID: 1789283, Thread 1789621: 
2025-07-17T11:07:25.6776646Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6776852Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6777053Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6777251Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6777747Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6777953Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6778157Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6778252Z SIGABRT(6), PID: 1789282, Thread 1793638: 
2025-07-17T11:07:25.6778719Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6778932Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6779231Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6779435Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6779889Z frame #4: <unknown function> + 0x5a02023 (0x7ff7f1065023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6780471Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff7f10691ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6781171Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff7f10601d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6781570Z frame #7: <unknown function> + 0x79d655 (0x7ff802550655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6781811Z frame #8: <unknown function> + 0xdc253 (0x7ff7b959b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6782028Z frame #9: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6782254Z frame #10: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6782258Z 
2025-07-17T11:07:25.6782262Z 
2025-07-17T11:07:25.6782354Z SIGABRT(6), PID: 1789276, Thread 1789297: 
2025-07-17T11:07:25.6782836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6783046Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6783257Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6783462Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6783983Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6784282Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6784497Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6784502Z 
2025-07-17T11:07:25.6784592Z SIGABRT(6), PID: 1789283, Thread 1789623: 
2025-07-17T11:07:25.6785072Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6785278Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6785482Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6785681Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6786186Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6786394Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6786607Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6786613Z 
2025-07-17T11:07:25.6786702Z SIGABRT(6), PID: 1789276, Thread 1789298: 
2025-07-17T11:07:25.6787174Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6787379Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6787650Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6787852Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6788411Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6788624Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6788834Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6788899Z 
2025-07-17T11:07:25.6788993Z SIGABRT(6), PID: 1789283, Thread 1789625: 
2025-07-17T11:07:25.6789464Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6789667Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6789875Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6790076Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6790571Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6790780Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6790988Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6790992Z 
2025-07-17T11:07:25.6791083Z SIGABRT(6), PID: 1789276, Thread 1789299: 
2025-07-17T11:07:25.6791556Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6791767Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6791971Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6792241Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6792743Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6792953Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6793162Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6793167Z 
2025-07-17T11:07:25.6793260Z SIGABRT(6), PID: 1789283, Thread 1789627: 
2025-07-17T11:07:25.6793727Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6793939Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6794144Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6794341Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6794831Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6795039Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6795245Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6795249Z 
2025-07-17T11:07:25.6795339Z SIGABRT(6), PID: 1789276, Thread 1789300: 
2025-07-17T11:07:25.6795870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6796148Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6796355Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6796557Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6797048Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6797320Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6797529Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6797533Z 
2025-07-17T11:07:25.6797627Z SIGABRT(6), PID: 1789283, Thread 1789629: 
2025-07-17T11:07:25.6798090Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6798300Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6798502Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6798699Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6799187Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6799395Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6799602Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6799608Z 
2025-07-17T11:07:25.6799701Z SIGABRT(6), PID: 1789283, Thread 1789631: 
2025-07-17T11:07:25.6800168Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6800445Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6800646Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6800842Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6801338Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6801542Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6801753Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6801843Z SIGABRT(6), PID: 1789276, Thread 1789301: 
2025-07-17T11:07:25.6802319Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6802524Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6802730Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6802928Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6803423Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6803629Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6804011Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6804018Z 
2025-07-17T11:07:25.6804021Z 
2025-07-17T11:07:25.6804193Z SIGABRT(6), PID: 1789283, Thread 1789633: 
2025-07-17T11:07:25.6804669Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6804876Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6805144Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6805342Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6805832Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6806041Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6806253Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6806341Z SIGABRT(6), PID: 1789282, Thread 1793639: 
2025-07-17T11:07:25.6806810Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6807019Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6807230Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6807428Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6807808Z frame #4: <unknown function> + 0x5a02023 (0x7ff7f1065023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6808378Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff7f10691ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6809043Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff7f10601d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6809423Z frame #7: <unknown function> + 0x79d655 (0x7ff802550655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6809660Z frame #8: <unknown function> + 0xdc253 (0x7ff7b959b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6809868Z frame #9: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6810087Z frame #10: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6810178Z SIGABRT(6), PID: 1789276, Thread 1789302: 
2025-07-17T11:07:25.6810654Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6810862Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6811068Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6811266Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6811767Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6811972Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6812183Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6812254Z 
2025-07-17T11:07:25.6812258Z 
2025-07-17T11:07:25.6812262Z 
2025-07-17T11:07:25.6812359Z SIGABRT(6), PID: 1789283, Thread 1789635: 
2025-07-17T11:07:25.6812884Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6813093Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6813297Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6813582Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6814074Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6814284Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6814492Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6814499Z 
2025-07-17T11:07:25.6814590Z SIGABRT(6), PID: 1789276, Thread 1789303: 
2025-07-17T11:07:25.6815058Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6815264Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6815470Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6815672Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6816214Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6816427Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6816726Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6816731Z 
2025-07-17T11:07:25.6816823Z SIGABRT(6), PID: 1789283, Thread 1789637: 
2025-07-17T11:07:25.6817291Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6817496Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6817701Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6817903Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6818394Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6818601Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6818809Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6818813Z 
2025-07-17T11:07:25.6818905Z SIGABRT(6), PID: 1789276, Thread 1789304: 
2025-07-17T11:07:25.6819373Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6819581Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6819782Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6819981Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6820551Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6820845Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6821056Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6821062Z 
2025-07-17T11:07:25.6821151Z SIGABRT(6), PID: 1789283, Thread 1789639: 
2025-07-17T11:07:25.6821627Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6821909Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6822114Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6822312Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6822814Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6823020Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6823230Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6823234Z 
2025-07-17T11:07:25.6823322Z SIGABRT(6), PID: 1789283, Thread 1789641: 
2025-07-17T11:07:25.6823796Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6823998Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6824205Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6824403Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6824967Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6825172Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6825382Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6825388Z 
2025-07-17T11:07:25.6825477Z SIGABRT(6), PID: 1789276, Thread 1789305: 
2025-07-17T11:07:25.6825941Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6826143Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6826350Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6826546Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6827045Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6827251Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6827461Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6827466Z 
2025-07-17T11:07:25.6827554Z SIGABRT(6), PID: 1789283, Thread 1789643: 
2025-07-17T11:07:25.6828019Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6828226Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6828496Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6828757Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6829258Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6829461Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6829735Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6829739Z 
2025-07-17T11:07:25.6829828Z SIGABRT(6), PID: 1789276, Thread 1789306: 
2025-07-17T11:07:25.6830298Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6830505Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6830714Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6830909Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6831408Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6831617Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6831830Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6831834Z 
2025-07-17T11:07:25.6831921Z SIGABRT(6), PID: 1789282, Thread 1793640: 
2025-07-17T11:07:25.6832391Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6832603Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6832880Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6833081Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6833457Z frame #4: <unknown function> + 0x5a02023 (0x7ff7f1065023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6834028Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff7f10691ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6834617Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff7f10601d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6834998Z frame #7: <unknown function> + 0x79d655 (0x7ff802550655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6835235Z frame #8: <unknown function> + 0xdc253 (0x7ff7b959b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6835448Z frame #9: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6835666Z frame #10: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6835671Z 
2025-07-17T11:07:25.6835763Z SIGABRT(6), PID: 1789283, Thread 1789645: 
2025-07-17T11:07:25.6836227Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6836434Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6836727Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6836931Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6837490Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6837701Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6837906Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6837973Z 
2025-07-17T11:07:25.6838068Z SIGABRT(6), PID: 1789276, Thread 1789307: 
2025-07-17T11:07:25.6838533Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6838744Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6838949Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6839151Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6839645Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6839855Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6840064Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6840068Z 
2025-07-17T11:07:25.6840163Z SIGABRT(6), PID: 1789283, Thread 1789648: 
2025-07-17T11:07:25.6840627Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6840835Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6841107Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6841305Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6841795Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6842005Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6842211Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6842220Z 
2025-07-17T11:07:25.6842310Z SIGABRT(6), PID: 1789276, Thread 1789308: 
2025-07-17T11:07:25.6842778Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6842983Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6843190Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6843387Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6843882Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6844090Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6844299Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6844304Z 
2025-07-17T11:07:25.6844391Z SIGABRT(6), PID: 1789283, Thread 1789650: 
2025-07-17T11:07:25.6844927Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6845197Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6845405Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6845601Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6846090Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6846361Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6846569Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6846573Z 
2025-07-17T11:07:25.6846661Z SIGABRT(6), PID: 1789276, Thread 1789309: 
2025-07-17T11:07:25.6847136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6847343Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6847550Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6847747Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6848243Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6848451Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6848658Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6848662Z 
2025-07-17T11:07:25.6848752Z SIGABRT(6), PID: 1789283, Thread 1789652: 
2025-07-17T11:07:25.6849289Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6849493Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6849697Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6849892Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6850384Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6850589Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6850800Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6850804Z 
2025-07-17T11:07:25.6850894Z SIGABRT(6), PID: 1789276, Thread 1789310: 
2025-07-17T11:07:25.6851367Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6851570Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6851775Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6851975Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6852470Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6852678Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6852955Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6852962Z 
2025-07-17T11:07:25.6853124Z SIGABRT(6), PID: 1789283, Thread 1789654: 
2025-07-17T11:07:25.6853594Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6853801Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6854064Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6854262Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6854754Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6854963Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6855174Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6855181Z 
2025-07-17T11:07:25.6855274Z SIGABRT(6), PID: 1789283, Thread 1789656: 
2025-07-17T11:07:25.6855737Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6855991Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6856198Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6856397Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6856886Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6857094Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6857387Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6857391Z 
2025-07-17T11:07:25.6857486Z SIGABRT(6), PID: 1789276, Thread 1789311: 
2025-07-17T11:07:25.6857951Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6858163Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6858366Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6858567Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6859058Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6859273Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6859479Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6859483Z 
2025-07-17T11:07:25.6859575Z SIGABRT(6), PID: 1789283, Thread 1789658: 
2025-07-17T11:07:25.6860036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6860246Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6860447Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6860644Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6861281Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6861495Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6861702Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6861706Z 
2025-07-17T11:07:25.6861797Z SIGABRT(6), PID: 1789276, Thread 1789312: 
2025-07-17T11:07:25.6862338Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6862547Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6862748Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6862952Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6863443Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6863653Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6863860Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6863952Z SIGABRT(6), PID: 1789282, Thread 1793641: 
2025-07-17T11:07:25.6864423Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6864629Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6864839Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6865036Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6865509Z frame #4: <unknown function> + 0x5a02023 (0x7ff7f1065023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6866073Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff7f10691ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6866667Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff7f10601d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6867048Z frame #7: <unknown function> + 0x79d655 (0x7ff802550655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6867284Z frame #8: <unknown function> + 0xdc253 (0x7ff7b959b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6867495Z frame #9: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6867719Z frame #10: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6867818Z SIGABRT(6), PID: 1789283, Thread 1789660: 
2025-07-17T11:07:25.6868292Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6868503Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6868710Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6868906Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6869474Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6869745Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6869958Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6869964Z 
2025-07-17T11:07:25.6869968Z 
2025-07-17T11:07:25.6869971Z 
2025-07-17T11:07:25.6870059Z SIGABRT(6), PID: 1789283, Thread 1789662: 
2025-07-17T11:07:25.6870530Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6870797Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6871004Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6871202Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6871706Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6871916Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6872125Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6872128Z 
2025-07-17T11:07:25.6872217Z SIGABRT(6), PID: 1789276, Thread 1789315: 
2025-07-17T11:07:25.6872688Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6872898Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6873100Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6873300Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6873863Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6874075Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6874281Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6874286Z 
2025-07-17T11:07:25.6874380Z SIGABRT(6), PID: 1789283, Thread 1789664: 
2025-07-17T11:07:25.6874843Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6875051Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6875255Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6875454Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6875945Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6876151Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6876356Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6876362Z 
2025-07-17T11:07:25.6876454Z SIGABRT(6), PID: 1789276, Thread 1789317: 
2025-07-17T11:07:25.6876916Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6877124Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6877393Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6877657Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6878150Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6878359Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6878635Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6878640Z 
2025-07-17T11:07:25.6878735Z SIGABRT(6), PID: 1789283, Thread 1789666: 
2025-07-17T11:07:25.6879203Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6879412Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6879618Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6879818Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6880303Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6880513Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6880719Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6880723Z 
2025-07-17T11:07:25.6880817Z SIGABRT(6), PID: 1789276, Thread 1789319: 
2025-07-17T11:07:25.6881281Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6881486Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6881759Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6881960Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6882450Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6882659Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6882867Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6882870Z 
2025-07-17T11:07:25.6882962Z SIGABRT(6), PID: 1789283, Thread 1789668: 
2025-07-17T11:07:25.6883433Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6883639Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6883843Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6884036Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6884529Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6884734Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6884943Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6885031Z SIGABRT(6), PID: 1789276, Thread 1789322: 
2025-07-17T11:07:25.6885567Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6885837Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6886044Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6886241Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6886801Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6887009Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6887221Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6887225Z 
2025-07-17T11:07:25.6887228Z 
2025-07-17T11:07:25.6887318Z SIGABRT(6), PID: 1789276, Thread 1789324: 
2025-07-17T11:07:25.6887791Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6887995Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6888198Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6888395Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6888885Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6889096Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6889308Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6889312Z 
2025-07-17T11:07:25.6889468Z SIGABRT(6), PID: 1789283, Thread 1789670: 
2025-07-17T11:07:25.6889941Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6890145Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6890349Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6890545Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6891038Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6891247Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6891461Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6891467Z 
2025-07-17T11:07:25.6891557Z SIGABRT(6), PID: 1789282, Thread 1793642: 
2025-07-17T11:07:25.6892022Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6892228Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6892439Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6892639Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6893016Z frame #4: <unknown function> + 0x5a02023 (0x7ff7f1065023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6893641Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff7f10691ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6894318Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff7f10601d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6894703Z frame #7: <unknown function> + 0x79d655 (0x7ff802550655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6895022Z frame #8: <unknown function> + 0xdc253 (0x7ff7b959b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6895237Z frame #9: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6895455Z frame #10: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6895460Z 
2025-07-17T11:07:25.6895553Z SIGABRT(6), PID: 1789276, Thread 1789325: 
2025-07-17T11:07:25.6896066Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6896276Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6896478Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6896676Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6897174Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6897384Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6897593Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6897599Z 
2025-07-17T11:07:25.6897692Z SIGABRT(6), PID: 1789283, Thread 1789673: 
2025-07-17T11:07:25.6898247Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6898453Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6898655Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6898856Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6899350Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6899561Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6899768Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6899773Z 
2025-07-17T11:07:25.6899868Z SIGABRT(6), PID: 1789276, Thread 1789328: 
2025-07-17T11:07:25.6900332Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6900542Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6900746Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6900950Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6901443Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6901737Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6901948Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6902028Z 
2025-07-17T11:07:25.6902124Z SIGABRT(6), PID: 1789283, Thread 1789675: 
2025-07-17T11:07:25.6902593Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6902801Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6903081Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6903281Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6903771Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6903982Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6904192Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6904199Z 
2025-07-17T11:07:25.6904287Z SIGABRT(6), PID: 1789283, Thread 1789677: 
2025-07-17T11:07:25.6904752Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6904957Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6905159Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6905353Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6905847Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6906125Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6906337Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6906341Z 
2025-07-17T11:07:25.6906430Z SIGABRT(6), PID: 1789276, Thread 1789331: 
2025-07-17T11:07:25.6906898Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6907102Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6907314Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6907509Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6908002Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6908213Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6908421Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6908426Z 
2025-07-17T11:07:25.6908513Z SIGABRT(6), PID: 1789283, Thread 1789679: 
2025-07-17T11:07:25.6908978Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6909180Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6909386Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6909654Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6910212Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6910421Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6910630Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6910723Z SIGABRT(6), PID: 1789276, Thread 1789333: 
2025-07-17T11:07:25.6911250Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6911456Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6911660Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6911859Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6912354Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6912560Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6912769Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6912775Z 
2025-07-17T11:07:25.6912778Z 
2025-07-17T11:07:25.6912865Z SIGABRT(6), PID: 1789283, Thread 1789681: 
2025-07-17T11:07:25.6913333Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6913536Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6913743Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6913937Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6914503Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6914712Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6914921Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6914926Z 
2025-07-17T11:07:25.6915014Z SIGABRT(6), PID: 1789276, Thread 1789336: 
2025-07-17T11:07:25.6915481Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6915687Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6915892Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6916096Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6916586Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6916795Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6917004Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6917008Z 
2025-07-17T11:07:25.6917102Z SIGABRT(6), PID: 1789283, Thread 1789683: 
2025-07-17T11:07:25.6917564Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6917842Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6918108Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6918310Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6918803Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6919074Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6919281Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6919285Z 
2025-07-17T11:07:25.6919376Z SIGABRT(6), PID: 1789276, Thread 1789339: 
2025-07-17T11:07:25.6919843Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6920051Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6920257Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6920457Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6920958Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6921170Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6921379Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6921383Z 
2025-07-17T11:07:25.6921475Z SIGABRT(6), PID: 1789283, Thread 1789685: 
2025-07-17T11:07:25.6921939Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6922223Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6922428Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6922629Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6923122Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6923334Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6923541Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6923545Z 
2025-07-17T11:07:25.6923637Z SIGABRT(6), PID: 1789282, Thread 1793643: 
2025-07-17T11:07:25.6924104Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6924317Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6924520Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6924719Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6925091Z frame #4: <unknown function> + 0x5a02023 (0x7ff7f1065023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6925662Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff7f10691ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6926400Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff7f10601d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6926785Z frame #7: <unknown function> + 0x79d655 (0x7ff802550655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6927023Z frame #8: <unknown function> + 0xdc253 (0x7ff7b959b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6927239Z frame #9: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6927538Z frame #10: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6927543Z 
2025-07-17T11:07:25.6927635Z SIGABRT(6), PID: 1789276, Thread 1789344: 
2025-07-17T11:07:25.6928121Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6928335Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6928546Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6928746Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6929249Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6929460Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6929672Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6929676Z 
2025-07-17T11:07:25.6929765Z SIGABRT(6), PID: 1789283, Thread 1789686: 
2025-07-17T11:07:25.6930241Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6930637Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6930845Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6931039Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6931540Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6931747Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6931964Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6931968Z 
2025-07-17T11:07:25.6932057Z SIGABRT(6), PID: 1789276, Thread 1789347: 
2025-07-17T11:07:25.6932536Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6932745Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6932953Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6933149Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6933647Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6933854Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6934069Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6934073Z 
2025-07-17T11:07:25.6934231Z SIGABRT(6), PID: 1789283, Thread 1789688: 
2025-07-17T11:07:25.6934765Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6934976Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6935183Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6935382Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6936003Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6936213Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6936420Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6936426Z 
2025-07-17T11:07:25.6936520Z SIGABRT(6), PID: 1789276, Thread 1789350: 
2025-07-17T11:07:25.6936989Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6937196Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6937398Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6937597Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6938086Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6938297Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6938505Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6938684Z SIGABRT(6), PID: 1789283, Thread 1789690: 
2025-07-17T11:07:25.6939155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6939360Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6939561Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6939761Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6940252Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6940461Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6940671Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6940677Z 
2025-07-17T11:07:25.6940681Z 
2025-07-17T11:07:25.6940776Z SIGABRT(6), PID: 1789283, Thread 1789692: 
2025-07-17T11:07:25.6941241Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6941452Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6941659Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6941859Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6942348Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6942640Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6942921Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6942926Z 
2025-07-17T11:07:25.6943019Z SIGABRT(6), PID: 1789276, Thread 1789352: 
2025-07-17T11:07:25.6943488Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6943771Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6943974Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6944172Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6944664Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6944876Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6945084Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6945088Z 
2025-07-17T11:07:25.6945181Z SIGABRT(6), PID: 1789283, Thread 1789693: 
2025-07-17T11:07:25.6945646Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6945852Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6946052Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6946253Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6946754Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6947030Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6947244Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6947248Z 
2025-07-17T11:07:25.6947338Z SIGABRT(6), PID: 1789276, Thread 1789359: 
2025-07-17T11:07:25.6947811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6948018Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6948222Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6948419Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6948919Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6949127Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6949337Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6949341Z 
2025-07-17T11:07:25.6949429Z SIGABRT(6), PID: 1789283, Thread 1789695: 
2025-07-17T11:07:25.6949902Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6950107Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6950309Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6950573Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6951130Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6951340Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6951551Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6951616Z 
2025-07-17T11:07:25.6951707Z SIGABRT(6), PID: 1789276, Thread 1789361: 
2025-07-17T11:07:25.6952180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6952385Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6952591Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6952788Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6953286Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6953492Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6953702Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6953707Z 
2025-07-17T11:07:25.6953798Z SIGABRT(6), PID: 1789282, Thread 1793644: 
2025-07-17T11:07:25.6954266Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6954471Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6954682Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6954955Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6955329Z frame #4: <unknown function> + 0x5a02023 (0x7ff7f1065023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6955898Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff7f10691ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6956495Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff7f10601d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6956880Z frame #7: <unknown function> + 0x79d655 (0x7ff802550655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6957113Z frame #8: <unknown function> + 0xdc253 (0x7ff7b959b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6957331Z frame #9: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6957550Z frame #10: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6957554Z 
2025-07-17T11:07:25.6957646Z SIGABRT(6), PID: 1789283, Thread 1789697: 
2025-07-17T11:07:25.6958123Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6958332Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6958536Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6958827Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6959390Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6959602Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6959809Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6959813Z 
2025-07-17T11:07:25.6959969Z SIGABRT(6), PID: 1789276, Thread 1789365: 
2025-07-17T11:07:25.6960437Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6960645Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6960848Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6961049Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6961546Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6961756Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6961964Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6961969Z 
2025-07-17T11:07:25.6962060Z SIGABRT(6), PID: 1789283, Thread 1789699: 
2025-07-17T11:07:25.6962525Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6962729Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6962933Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6963203Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6963691Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6963904Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6964112Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6964116Z 
2025-07-17T11:07:25.6964206Z SIGABRT(6), PID: 1789276, Thread 1789369: 
2025-07-17T11:07:25.6964669Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6964878Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6965081Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6965285Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6965779Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6965987Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6966195Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6966204Z 
2025-07-17T11:07:25.6966292Z SIGABRT(6), PID: 1789283, Thread 1789701: 
2025-07-17T11:07:25.6966756Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6967025Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6967294Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6967492Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6967992Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6968258Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6968472Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6968476Z 
2025-07-17T11:07:25.6968566Z SIGABRT(6), PID: 1789276, Thread 1789371: 
2025-07-17T11:07:25.6969036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6969248Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6969454Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6969649Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6970141Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6970350Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6970562Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6970567Z 
2025-07-17T11:07:25.6970656Z SIGABRT(6), PID: 1789283, Thread 1789703: 
2025-07-17T11:07:25.6971136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6971412Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6971618Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6971812Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6972310Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6972516Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6972726Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6972730Z 
2025-07-17T11:07:25.6972822Z SIGABRT(6), PID: 1789276, Thread 1789376: 
2025-07-17T11:07:25.6973291Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6973497Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6973702Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6973899Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6974400Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6974605Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6974814Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6974985Z SIGABRT(6), PID: 1789283, Thread 1789705: 
2025-07-17T11:07:25.6975521Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6975726Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6975980Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6976254Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6976751Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6976956Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6977169Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6977173Z 
2025-07-17T11:07:25.6977178Z 
2025-07-17T11:07:25.6977269Z SIGABRT(6), PID: 1789283, Thread 1789707: 
2025-07-17T11:07:25.6977738Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6977943Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6978145Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6978343Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6978836Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6979043Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6979252Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6979337Z 
2025-07-17T11:07:25.6979430Z SIGABRT(6), PID: 1789276, Thread 1789379: 
2025-07-17T11:07:25.6979899Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6980104Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6980306Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6980506Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6980999Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6981213Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6981421Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6981426Z 
2025-07-17T11:07:25.6981522Z SIGABRT(6), PID: 1789276, Thread 1789382: 
2025-07-17T11:07:25.6981988Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6982199Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6982401Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6982599Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6983164Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6983452Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6983663Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6983667Z 
2025-07-17T11:07:25.6983760Z SIGABRT(6), PID: 1789283, Thread 1789709: 
2025-07-17T11:07:25.6984226Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6984495Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6984696Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6984892Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6985386Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6985597Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6985803Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6985896Z SIGABRT(6), PID: 1789282, Thread 1793645: 
2025-07-17T11:07:25.6986362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff7bb32eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6986573Z frame #1: <unknown function> + 0x42520 (0x7ff8acbe9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6986774Z frame #2: <unknown function> + 0x91117 (0x7ff8acc38117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6986975Z frame #3: pthread_cond_wait + 0x211 (0x7ff8acc3aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6987349Z frame #4: <unknown function> + 0x5a02023 (0x7ff7f1065023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6988009Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff7f10691ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6988597Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff7f10601d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.6988985Z frame #7: <unknown function> + 0x79d655 (0x7ff802550655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.6989219Z frame #8: <unknown function> + 0xdc253 (0x7ff7b959b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.6989428Z frame #9: <unknown function> + 0x94ac3 (0x7ff8acc3bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6989648Z frame #10: <unknown function> + 0x126850 (0x7ff8acccd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6989655Z 
2025-07-17T11:07:25.6989659Z 
2025-07-17T11:07:25.6989747Z SIGABRT(6), PID: 1789276, Thread 1789385: 
2025-07-17T11:07:25.6990220Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6990426Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6990635Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6990834Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6991400Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6991613Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6991884Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6991889Z 
2025-07-17T11:07:25.6991980Z SIGABRT(6), PID: 1789283, Thread 1789711: 
2025-07-17T11:07:25.6992454Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6992718Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6992926Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6993119Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6993620Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6993826Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6994037Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6994044Z 
2025-07-17T11:07:25.6994131Z SIGABRT(6), PID: 1789276, Thread 1789387: 
2025-07-17T11:07:25.6994604Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6994812Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6995018Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6995215Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6995709Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6995985Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6996196Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6996200Z 
2025-07-17T11:07:25.6996289Z SIGABRT(6), PID: 1789283, Thread 1789713: 
2025-07-17T11:07:25.6996761Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6996966Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6997170Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6997367Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6997864Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.6998074Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6998281Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6998286Z 
2025-07-17T11:07:25.6998379Z SIGABRT(6), PID: 1789283, Thread 1789715: 
2025-07-17T11:07:25.6998844Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.6999049Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6999254Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.6999517Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7000076Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7000288Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7000497Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7000652Z SIGABRT(6), PID: 1789276, Thread 1789390: 
2025-07-17T11:07:25.7001122Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7001328Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7001532Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7001731Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7002231Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7002443Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7002651Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7002657Z 
2025-07-17T11:07:25.7002660Z 
2025-07-17T11:07:25.7002753Z SIGABRT(6), PID: 1789283, Thread 1789717: 
2025-07-17T11:07:25.7003216Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7003423Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7003625Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7003895Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7004385Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7004594Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7004801Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7004892Z SIGABRT(6), PID: 1789276, Thread 1789393: 
2025-07-17T11:07:25.7005358Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7005570Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7005776Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7005976Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7006469Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7006677Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7006884Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7006889Z 
2025-07-17T11:07:25.7006892Z 
2025-07-17T11:07:25.7006982Z SIGABRT(6), PID: 1789276, Thread 1789396: 
2025-07-17T11:07:25.7007519Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7007788Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7007994Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7008196Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7008689Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7008960Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7009170Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7009260Z SIGABRT(6), PID: 1789283, Thread 1789718: 
2025-07-17T11:07:25.7009731Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7009937Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7010139Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7010334Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7010828Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7011034Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7011243Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7011247Z 
2025-07-17T11:07:25.7011253Z 
2025-07-17T11:07:25.7011344Z SIGABRT(6), PID: 1789276, Thread 1789398: 
2025-07-17T11:07:25.7011818Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7012096Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7012301Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7012498Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7012999Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7013204Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7013416Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7013507Z SIGABRT(6), PID: 1789283, Thread 1789720: 
2025-07-17T11:07:25.7013977Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7014179Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7014382Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7014581Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7015077Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7015282Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7015563Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7015568Z 
2025-07-17T11:07:25.7015574Z 
2025-07-17T11:07:25.7015728Z SIGABRT(6), PID: 1789283, Thread 1789722: 
2025-07-17T11:07:25.7016253Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7016458Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7016664Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7016961Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7017462Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7017668Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7017879Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7017887Z 
2025-07-17T11:07:25.7017977Z SIGABRT(6), PID: 1789276, Thread 1789401: 
2025-07-17T11:07:25.7018450Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7018654Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7018861Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7019060Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7019554Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7019767Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7020063Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7020068Z 
2025-07-17T11:07:25.7020161Z SIGABRT(6), PID: 1789283, Thread 1789723: 
2025-07-17T11:07:25.7020627Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7020833Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7021040Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7021235Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7021730Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7021941Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7022148Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7022152Z 
2025-07-17T11:07:25.7022244Z SIGABRT(6), PID: 1789276, Thread 1789403: 
2025-07-17T11:07:25.7022706Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7022917Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7023121Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7023320Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7023891Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7024180Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7024391Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7024395Z 
2025-07-17T11:07:25.7024486Z SIGABRT(6), PID: 1789283, Thread 1789724: 
2025-07-17T11:07:25.7024951Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7025224Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7025425Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7025624Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7026117Z frame #4: <unknown function> + 0x3475ab (0x7f897384f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7026327Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7026533Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7026538Z 
2025-07-17T11:07:25.7026629Z SIGABRT(6), PID: 1789276, Thread 1789406: 
2025-07-17T11:07:25.7027096Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7027301Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7027502Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7027704Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7028270Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7028484Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7028690Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7028695Z 
2025-07-17T11:07:25.7028788Z SIGABRT(6), PID: 1789283, Thread 1789806: 
2025-07-17T11:07:25.7029254Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7029706Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f88850ac08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7029916Z frame #2: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7030109Z frame #3: pthread_kill + 0x12c (0x7f89769b99fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7030275Z frame #4: raise + 0x16 (0x7f8976965476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7030434Z frame #5: abort + 0xd3 (0x7f897694b7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7030676Z frame #6: <unknown function> + 0x972cc (0x7f87f2c352cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7030910Z frame #7: <unknown function> + 0x95f47 (0x7f87f2c33f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7031137Z frame #8: <unknown function> + 0x34b71 (0x7f87f2bd2b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7031349Z frame #9: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7031631Z frame #10: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7031641Z 
2025-07-17T11:07:25.7031736Z SIGABRT(6), PID: 1789276, Thread 1789409: 
2025-07-17T11:07:25.7032269Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7032483Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7032692Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7032952Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7033456Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7033663Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7033875Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7033881Z 
2025-07-17T11:07:25.7033972Z SIGABRT(6), PID: 1789283, Thread 1792918: 
2025-07-17T11:07:25.7034446Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7034648Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7034818Z frame #2: __poll + 0x4f (0x7f8976a3bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7035217Z frame #3: <unknown function> + 0x489f (0x7f897654789f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.7035448Z frame #4: <unknown function> + 0x1483bc (0x55af30a103bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7035696Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7035926Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7036236Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7036465Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7036709Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55af309f8a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7036940Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7037186Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55af309f6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7037416Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7037657Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7037882Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7038122Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7038345Z frame #16: <unknown function> + 0x147734 (0x55af30a0f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7038565Z frame #17: <unknown function> + 0x21ca82 (0x55af30ae4a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7038789Z frame #18: <unknown function> + 0x21c9e4 (0x55af30ae49e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7039000Z frame #19: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7039213Z frame #20: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7039217Z 
2025-07-17T11:07:25.7039308Z SIGABRT(6), PID: 1789276, Thread 1789412: 
2025-07-17T11:07:25.7039854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7040128Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7040339Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7040541Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7041104Z frame #4: <unknown function> + 0x3475ab (0x7fd31f7435ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7041313Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7041524Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7041528Z 
2025-07-17T11:07:25.7041621Z SIGABRT(6), PID: 1789276, Thread 1789800: 
2025-07-17T11:07:25.7042099Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7042554Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fd230fa008a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7042764Z frame #2: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7042959Z frame #3: pthread_kill + 0x12c (0x7fd3228ad9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7043123Z frame #4: raise + 0x16 (0x7fd322859476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7043282Z frame #5: abort + 0xd3 (0x7fd32283f7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7043516Z frame #6: <unknown function> + 0x972cc (0x7fd19eb292cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7043744Z frame #7: <unknown function> + 0x95f47 (0x7fd19eb27f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7044052Z frame #8: <unknown function> + 0x34b71 (0x7fd19eac6b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7044260Z frame #9: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7044475Z frame #10: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7044480Z 
2025-07-17T11:07:25.7044570Z SIGABRT(6), PID: 1789283, Thread 1793520: 
2025-07-17T11:07:25.7045045Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7045249Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7045454Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7045675Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f89769b72dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7046119Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f88b0df4048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7046546Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f88b0df6bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7046774Z frame #6: <unknown function> + 0xdc253 (0x7f8883318253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7046983Z frame #7: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7047193Z frame #8: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7047197Z 
2025-07-17T11:07:25.7047285Z SIGABRT(6), PID: 1789276, Thread 1793316: 
2025-07-17T11:07:25.7047833Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7048126Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7048296Z frame #2: __poll + 0x4f (0x7fd32292fbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7048699Z frame #3: <unknown function> + 0x489f (0x7fd32243b89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.7048989Z frame #4: <unknown function> + 0x1483bc (0x55b9e29b23bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7049231Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7049458Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7049696Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7049922Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7050170Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55b9e299aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7050396Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7050646Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b9e2998c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7050871Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7051111Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7051335Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7051570Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7051791Z frame #16: <unknown function> + 0x147734 (0x55b9e29b1734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7052103Z frame #17: <unknown function> + 0x21ca82 (0x55b9e2a86a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7052322Z frame #18: <unknown function> + 0x21c9e4 (0x55b9e2a869e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7052536Z frame #19: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7052747Z frame #20: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7052753Z 
2025-07-17T11:07:25.7052846Z SIGABRT(6), PID: 1789283, Thread 1793521: 
2025-07-17T11:07:25.7053316Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7053525Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7053732Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7053960Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f89769b72dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7054430Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f88b0dee741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7054657Z frame #5: <unknown function> + 0xdc253 (0x7f8883318253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7054866Z frame #6: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7055079Z frame #7: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7055083Z 
2025-07-17T11:07:25.7055173Z SIGABRT(6), PID: 1789283, Thread 1793522: 
2025-07-17T11:07:25.7055707Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7056036Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7056210Z frame #2: ioctl + 0x3f (0x7f8976a3d94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7056446Z frame #3: <unknown function> + 0x13eef0 (0x7f87f2cdcef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7056675Z frame #4: <unknown function> + 0x1378c4 (0x7f87f2cd58c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7057113Z frame #5: <unknown function> + 0x95b2b (0x7f87f2c33b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7057341Z frame #6: <unknown function> + 0x34b71 (0x7f87f2bd2b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7057549Z frame #7: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7057760Z frame #8: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7057766Z 
2025-07-17T11:07:25.7057856Z SIGABRT(6), PID: 1789283, Thread 1793552: 
2025-07-17T11:07:25.7058332Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7058536Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7058701Z frame #2: __poll + 0x4f (0x7f8976a3bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7058916Z frame #3: <unknown function> + 0xd4540 (0x7f883ae4c540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7059126Z frame #4: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7059330Z frame #5: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7059334Z 
2025-07-17T11:07:25.7059425Z SIGABRT(6), PID: 1789283, Thread 1793560: 
2025-07-17T11:07:25.7059892Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7060189Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7060347Z frame #2: __poll + 0x4f (0x7f8976a3bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7060565Z frame #3: <unknown function> + 0xd65a2 (0x7f883ae4e5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7060773Z frame #4: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7060983Z frame #5: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7060987Z 
2025-07-17T11:07:25.7061077Z SIGABRT(6), PID: 1789283, Thread 1793573: 
2025-07-17T11:07:25.7061563Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7061768Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7061979Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7062178Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7062394Z frame #4: <unknown function> + 0xd1e7f (0x7f883ae49e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7062601Z frame #5: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7062808Z frame #6: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7062812Z 
2025-07-17T11:07:25.7062901Z SIGABRT(6), PID: 1789283, Thread 1793576: 
2025-07-17T11:07:25.7063450Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7063659Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7063928Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7064136Z frame #3: <unknown function> + 0x9ca53 (0x7f89769bfa53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7064395Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55af309dcbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7064685Z frame #5: <unknown function> + 0x1d314f (0x55af30a9b14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7064908Z frame #6: <unknown function> + 0x1d2f6b (0x55af30a9af6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7065130Z frame #7: <unknown function> + 0x6df36 (0x55af30935f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7065370Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7065596Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7065842Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7066070Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7066307Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7066530Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7066767Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7066990Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7067225Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7067446Z frame #17: <unknown function> + 0x147734 (0x55af30a0f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7067742Z frame #18: <unknown function> + 0x21ca82 (0x55af30ae4a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7067965Z frame #19: <unknown function> + 0x21c9e4 (0x55af30ae49e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7068177Z frame #20: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7068388Z frame #21: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7068394Z 
2025-07-17T11:07:25.7068486Z SIGABRT(6), PID: 1789276, Thread 1793317: 
2025-07-17T11:07:25.7068962Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7069177Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7069361Z frame #2: epoll_wait + 0x5e (0x7fd32293ce2e in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7069703Z frame #3: uv__io_poll + 0x6bd (0x7fd2734f644d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7070022Z frame #4: uv_run + 0x12a (0x7fd2734ea36a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7070400Z frame #5: <unknown function> + 0x658a08a (0x7fd26785e08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7070632Z frame #6: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7070845Z frame #7: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7071054Z frame #8: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7071061Z 
2025-07-17T11:07:25.7071216Z SIGABRT(6), PID: 1789283, Thread 1793587: 
2025-07-17T11:07:25.7071749Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7071960Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7072127Z frame #2: read + 0x4c (0x7f8976a3781c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7072307Z frame #3: _Py_read + 0x4c (0x55af30aa2a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7072604Z frame #4: <unknown function> + 0x208c3b (0x55af30ad0c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7072825Z frame #5: <unknown function> + 0x14648d (0x55af30a0e48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7073073Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55af30a1d215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7073297Z frame #7: <unknown function> + 0x208ae7 (0x55af30ad0ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7073519Z frame #8: <unknown function> + 0x208472 (0x55af30ad0472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7073743Z frame #9: <unknown function> + 0x1da6ee (0x55af30aa26ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7073968Z frame #10: <unknown function> + 0x1483bc (0x55af30a103bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7074207Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7074437Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7074674Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55af309f43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7074896Z frame #14: <unknown function> + 0x147734 (0x55af30a0f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7075142Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55af309f6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7075370Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7075678Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7075906Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55af30a040ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7076139Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55af309f47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7076360Z frame #20: <unknown function> + 0x147734 (0x55af30a0f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7076579Z frame #21: <unknown function> + 0x21ca82 (0x55af30ae4a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7076799Z frame #22: <unknown function> + 0x21c9e4 (0x55af30ae49e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7077005Z frame #23: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7077223Z frame #24: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7077229Z 
2025-07-17T11:07:25.7077320Z SIGABRT(6), PID: 1789276, Thread 1793537: 
2025-07-17T11:07:25.7077800Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7078010Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7078217Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7078440Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fd3228ab2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7078884Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fd25cce8048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7079394Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fd25cceabf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7079693Z frame #6: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7079906Z frame #7: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7080120Z frame #8: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7080124Z 
2025-07-17T11:07:25.7080282Z SIGABRT(6), PID: 1789276, Thread 1793538: 
2025-07-17T11:07:25.7080759Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7080965Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7081174Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7081399Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fd3228ab2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7081877Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fd25cce2741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7082105Z frame #5: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7082314Z frame #6: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7082523Z frame #7: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7082527Z 
2025-07-17T11:07:25.7082618Z SIGABRT(6), PID: 1789276, Thread 1793540: 
2025-07-17T11:07:25.7083084Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7083296Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7083534Z frame #2: ioctl + 0x3f (0x7fd32293194f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7083773Z frame #3: <unknown function> + 0x13eef0 (0x7fd19ebd0ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7084004Z frame #4: <unknown function> + 0x1378c4 (0x7fd19ebc98c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7084235Z frame #5: <unknown function> + 0x95b2b (0x7fd19eb27b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7084463Z frame #6: <unknown function> + 0x34b71 (0x7fd19eac6b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7084675Z frame #7: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7084884Z frame #8: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7084976Z SIGABRT(6), PID: 1789283, Thread 1793614: 
2025-07-17T11:07:25.7085442Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7085655Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7085858Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7086060Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7086436Z frame #4: <unknown function> + 0x5a02023 (0x7f88bade2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7087010Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f88bade61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7087674Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f88baddd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7088129Z frame #7: <unknown function> + 0x79d655 (0x7f88cc2cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7088358Z frame #8: <unknown function> + 0xdc253 (0x7f8883318253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7088566Z frame #9: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7088841Z frame #10: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7088850Z 
2025-07-17T11:07:25.7088854Z 
2025-07-17T11:07:25.7088943Z SIGABRT(6), PID: 1789276, Thread 1793557: 
2025-07-17T11:07:25.7089421Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7089629Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7089800Z frame #2: __poll + 0x4f (0x7fd32292fbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7090012Z frame #3: <unknown function> + 0xd4540 (0x7fd1e6d40540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7090221Z frame #4: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7090427Z frame #5: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7090434Z 
2025-07-17T11:07:25.7090524Z SIGABRT(6), PID: 1789276, Thread 1793565: 
2025-07-17T11:07:25.7090990Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7091198Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7091361Z frame #2: __poll + 0x4f (0x7fd32292fbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7091649Z frame #3: <unknown function> + 0xd65a2 (0x7fd1e6d425a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7091856Z frame #4: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7092063Z frame #5: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7092066Z 
2025-07-17T11:07:25.7092155Z SIGABRT(6), PID: 1789276, Thread 1793570: 
2025-07-17T11:07:25.7092628Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7092833Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7093037Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7093240Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7093460Z frame #4: <unknown function> + 0xd1e7f (0x7fd1e6d3de7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7093667Z frame #5: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7093874Z frame #6: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7093878Z 
2025-07-17T11:07:25.7093966Z SIGABRT(6), PID: 1789276, Thread 1793580: 
2025-07-17T11:07:25.7094436Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7094640Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7094845Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7095120Z frame #3: <unknown function> + 0x9ca53 (0x7fd3228b3a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7095439Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55b9e297ebc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7095673Z frame #5: <unknown function> + 0x1d314f (0x55b9e2a3d14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7095899Z frame #6: <unknown function> + 0x1d2f6b (0x55b9e2a3cf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7096178Z frame #7: <unknown function> + 0x6df36 (0x55b9e28d7f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7096503Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7096730Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7096972Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7097199Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7097438Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7097666Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7097899Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7098122Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7098357Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7098577Z frame #17: <unknown function> + 0x147734 (0x55b9e29b1734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7098802Z frame #18: <unknown function> + 0x21ca82 (0x55b9e2a86a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7099021Z frame #19: <unknown function> + 0x21c9e4 (0x55b9e2a869e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7099234Z frame #20: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7099530Z frame #21: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7099534Z 
2025-07-17T11:07:25.7099629Z SIGABRT(6), PID: 1789276, Thread 1793597: 
2025-07-17T11:07:25.7100100Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7100317Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7100478Z frame #2: read + 0x4c (0x7fd32292b81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7100660Z frame #3: _Py_read + 0x4c (0x55b9e2a44a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7100884Z frame #4: <unknown function> + 0x208c3b (0x55b9e2a72c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7105869Z frame #5: <unknown function> + 0x14648d (0x55b9e29b048d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7106187Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55b9e29bf215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7106441Z frame #7: <unknown function> + 0x208ae7 (0x55b9e2a72ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7106675Z frame #8: <unknown function> + 0x208472 (0x55b9e2a72472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7106908Z frame #9: <unknown function> + 0x1da6ee (0x55b9e2a446ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7107142Z frame #10: <unknown function> + 0x1483bc (0x55b9e29b23bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7107403Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7107633Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7108001Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55b9e29963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7108309Z frame #14: <unknown function> + 0x147734 (0x55b9e29b1734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7108568Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b9e2998c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7108796Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7109036Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7109338Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55b9e29a60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7109578Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55b9e29967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7109801Z frame #20: <unknown function> + 0x147734 (0x55b9e29b1734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7110027Z frame #21: <unknown function> + 0x21ca82 (0x55b9e2a86a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7110250Z frame #22: <unknown function> + 0x21c9e4 (0x55b9e2a869e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7110476Z frame #23: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7110692Z frame #24: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7110698Z 
2025-07-17T11:07:25.7110799Z SIGABRT(6), PID: 1789283, Thread 1793615: 
2025-07-17T11:07:25.7111301Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7111519Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7111727Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7111938Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7112419Z frame #4: <unknown function> + 0x5a02023 (0x7f88bade2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7112990Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f88bade61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7113588Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f88baddd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7113980Z frame #7: <unknown function> + 0x79d655 (0x7f88cc2cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7114214Z frame #8: <unknown function> + 0xdc253 (0x7f8883318253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7114432Z frame #9: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7114654Z frame #10: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7114659Z 
2025-07-17T11:07:25.7114762Z SIGABRT(6), PID: 1789276, Thread 1793654: 
2025-07-17T11:07:25.7115242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7115457Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7115666Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7115870Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7116313Z frame #4: <unknown function> + 0x5a02023 (0x7fd266cd6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7116940Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fd266cda1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7117537Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fd266cd11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7117988Z frame #7: <unknown function> + 0x79d655 (0x7fd2781c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7118223Z frame #8: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7118433Z frame #9: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7118651Z frame #10: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7118656Z 
2025-07-17T11:07:25.7118751Z SIGABRT(6), PID: 1789283, Thread 1793616: 
2025-07-17T11:07:25.7119227Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7119433Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7119640Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7119841Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7120215Z frame #4: <unknown function> + 0x5a02023 (0x7f88bade2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7120778Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f88bade61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7121446Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f88baddd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7121826Z frame #7: <unknown function> + 0x79d655 (0x7f88cc2cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7122056Z frame #8: <unknown function> + 0xdc253 (0x7f8883318253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7122265Z frame #9: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7122479Z frame #10: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7122483Z 
2025-07-17T11:07:25.7122574Z SIGABRT(6), PID: 1789276, Thread 1793655: 
2025-07-17T11:07:25.7123048Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7123257Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7123464Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7123663Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7124035Z frame #4: <unknown function> + 0x5a02023 (0x7fd266cd6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7124594Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fd266cda1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7125260Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fd266cd11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7125708Z frame #7: <unknown function> + 0x79d655 (0x7fd2781c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7125945Z frame #8: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7126161Z frame #9: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7126435Z frame #10: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7126440Z 
2025-07-17T11:07:25.7126533Z SIGABRT(6), PID: 1789283, Thread 1793617: 
2025-07-17T11:07:25.7127006Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7127217Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7127422Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7127627Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7127996Z frame #4: <unknown function> + 0x5a02023 (0x7f88bade2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7128555Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f88bade61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7129143Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f88baddd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7129525Z frame #7: <unknown function> + 0x79d655 (0x7f88cc2cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7129825Z frame #8: <unknown function> + 0xdc253 (0x7f8883318253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7130036Z frame #9: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7130246Z frame #10: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7130250Z 
2025-07-17T11:07:25.7130343Z SIGABRT(6), PID: 1789276, Thread 1793656: 
2025-07-17T11:07:25.7130815Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7131024Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7131227Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7131431Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7131802Z frame #4: <unknown function> + 0x5a02023 (0x7fd266cd6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7132361Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fd266cda1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7132948Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fd266cd11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7133328Z frame #7: <unknown function> + 0x79d655 (0x7fd2781c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7133556Z frame #8: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7133833Z frame #9: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7134112Z frame #10: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7134210Z SIGABRT(6), PID: 1789283, Thread 1793618: 
2025-07-17T11:07:25.7134683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7134973Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7135179Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7135378Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7135750Z frame #4: <unknown function> + 0x5a02023 (0x7f88bade2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7136396Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f88bade61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7136997Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f88baddd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7137378Z frame #7: <unknown function> + 0x79d655 (0x7f88cc2cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7137607Z frame #8: <unknown function> + 0xdc253 (0x7f8883318253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7137813Z frame #9: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7138025Z frame #10: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7138031Z 
2025-07-17T11:07:25.7138034Z 
2025-07-17T11:07:25.7138214Z SIGABRT(6), PID: 1789283, Thread 1793619: 
2025-07-17T11:07:25.7138689Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7138896Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7139103Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7139301Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7139674Z frame #4: <unknown function> + 0x5a02023 (0x7f88bade2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7140233Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f88bade61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7140825Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f88baddd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7141202Z frame #7: <unknown function> + 0x79d655 (0x7f88cc2cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7141429Z frame #8: <unknown function> + 0xdc253 (0x7f8883318253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7141636Z frame #9: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7141849Z frame #10: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7141853Z 
2025-07-17T11:07:25.7141941Z SIGABRT(6), PID: 1789283, Thread 1793620: 
2025-07-17T11:07:25.7142488Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7142771Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7142980Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7143176Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7143544Z frame #4: <unknown function> + 0x5a02023 (0x7f88bade2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7144205Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f88bade61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7144795Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f88baddd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7145175Z frame #7: <unknown function> + 0x79d655 (0x7f88cc2cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7145399Z frame #8: <unknown function> + 0xdc253 (0x7f8883318253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7145606Z frame #9: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7145818Z frame #10: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7145823Z 
2025-07-17T11:07:25.7146109Z Memory access fault by GPU node-7 (Agent handle: 0x558f7baf0160) on address 0x7f1d2ea1f000. Reason: Unknown.
2025-07-17T11:07:25.7146201Z SIGABRT(6), PID: 1789283, Thread 1793621: 
2025-07-17T11:07:25.7146680Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f88850abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7146964Z frame #1: <unknown function> + 0x42520 (0x7f8976965520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7147169Z frame #2: <unknown function> + 0x91117 (0x7f89769b4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7147370Z frame #3: pthread_cond_wait + 0x211 (0x7f89769b6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7147742Z frame #4: <unknown function> + 0x5a02023 (0x7f88bade2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7148306Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f88bade61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7148897Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f88baddd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7149276Z frame #7: <unknown function> + 0x79d655 (0x7f88cc2cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7149504Z frame #8: <unknown function> + 0xdc253 (0x7f8883318253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7149709Z frame #9: <unknown function> + 0x94ac3 (0x7f89769b7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7149922Z frame #10: <unknown function> + 0x126850 (0x7f8976a49850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7149927Z 
2025-07-17T11:07:25.7150017Z SIGABRT(6), PID: 1789276, Thread 1793657: 
2025-07-17T11:07:25.7150484Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7150693Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7150968Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7151232Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7151607Z frame #4: <unknown function> + 0x5a02023 (0x7fd266cd6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7152173Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fd266cda1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7152820Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fd266cd11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7153199Z frame #7: <unknown function> + 0x79d655 (0x7fd2781c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7153428Z frame #8: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7153642Z frame #9: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7153856Z frame #10: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7153860Z 
2025-07-17T11:07:25.7153951Z SIGABRT(6), PID: 1789276, Thread 1793658: 
2025-07-17T11:07:25.7154419Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7154630Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7154833Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7155033Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7155406Z frame #4: <unknown function> + 0x5a02023 (0x7fd266cd6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7156035Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fd266cda1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7156620Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fd266cd11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7156999Z frame #7: <unknown function> + 0x79d655 (0x7fd2781c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7157223Z frame #8: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7157434Z frame #9: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7157646Z frame #10: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7157653Z 
2025-07-17T11:07:25.7157745Z SIGABRT(6), PID: 1789276, Thread 1793659: 
2025-07-17T11:07:25.7158210Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7158417Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7158621Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7158822Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7159186Z frame #4: <unknown function> + 0x5a02023 (0x7fd266cd6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7159870Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fd266cda1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7160478Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fd266cd11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7160866Z frame #7: <unknown function> + 0x79d655 (0x7fd2781c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7161163Z frame #8: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7161383Z frame #9: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7161599Z frame #10: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7161607Z 
2025-07-17T11:07:25.7161700Z SIGABRT(6), PID: 1789276, Thread 1793660: 
2025-07-17T11:07:25.7162185Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7162394Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7162600Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7162800Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7163176Z frame #4: <unknown function> + 0x5a02023 (0x7fd266cd6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7163731Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fd266cda1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7164330Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fd266cd11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7164782Z frame #7: <unknown function> + 0x79d655 (0x7fd2781c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7165015Z frame #8: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7165227Z frame #9: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7165442Z frame #10: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7165447Z 
2025-07-17T11:07:25.7165538Z SIGABRT(6), PID: 1789276, Thread 1793661: 
2025-07-17T11:07:25.7166017Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fd230f9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7166225Z frame #1: <unknown function> + 0x42520 (0x7fd322859520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7166432Z frame #2: <unknown function> + 0x91117 (0x7fd3228a8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7166632Z frame #3: pthread_cond_wait + 0x211 (0x7fd3228aaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7167004Z frame #4: <unknown function> + 0x5a02023 (0x7fd266cd6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7167563Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fd266cda1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7168221Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fd266cd11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7168660Z frame #7: <unknown function> + 0x79d655 (0x7fd2781c1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7168897Z frame #8: <unknown function> + 0xdc253 (0x7fd22f20c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7169108Z frame #9: <unknown function> + 0x94ac3 (0x7fd3228abac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7169324Z frame #10: <unknown function> + 0x126850 (0x7fd32293d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7169389Z 
2025-07-17T11:07:25.7169480Z SIGABRT(6), PID: 1789277, Thread 1789277: 
2025-07-17T11:07:25.7169956Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7170162Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7170398Z frame #2: <unknown function> + 0x77435 (0x7f9c436b4435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7170633Z frame #3: <unknown function> + 0x76c8e (0x7f9c436b3c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7170860Z frame #4: <unknown function> + 0x6ad69 (0x7f9c436a7d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7171075Z frame #5: <unknown function> + 0x3f54db (0x7f9cd430c4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7171290Z frame #6: <unknown function> + 0x3e11d8 (0x7f9cd42f81d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7171503Z frame #7: <unknown function> + 0xb2f86 (0x7f9cd3fc9f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7171711Z frame #8: <unknown function> + 0x8e501 (0x7f9cd3fa5501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7172093Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f9cfedac7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.7172480Z frame #10: <unknown function> + 0xd307d9 (0x7f9d1d2fb7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7172780Z frame #11: <unknown function> + 0x139902 (0x55f249dc2902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7173035Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55f249db9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7173263Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7173506Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55f249db9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7173731Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7173975Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7174198Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7174439Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7174663Z frame #19: <unknown function> + 0x1474b2 (0x55f249dd04b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7174868Z frame #20: PyObject_Call + 0xbc (0x55f249dd0e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7175092Z frame #21: <unknown function> + 0x1ff69d (0x55f249e8869d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7175322Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55f249dbe023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7175567Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55f249db9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7175795Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7176081Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55f249db53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7176391Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7176729Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55f249db53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7176958Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7177198Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55f249db9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7177458Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55f249dbd4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7177758Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55f249dceab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7177993Z frame #32: <unknown function> + 0x205949 (0x55f249e8e949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7178228Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55f249dbe023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7178483Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55f249db9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7178716Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7178960Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55f249db53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7179181Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7179420Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55f249db53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7179643Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7179882Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55f249db53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7180104Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7180349Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55f249db63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7180652Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7181057Z frame #44: <unknown function> + 0x7ab3f7 (0x7f9d1cd763f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7181277Z frame #45: <unknown function> + 0x13bc88 (0x55f249dc4c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7181485Z frame #46: PyObject_Call + 0x20f (0x55f249dd0f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7181728Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55f249dba98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7181951Z frame #48: <unknown function> + 0x147608 (0x55f249dd0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7182191Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f249db7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7182421Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7182660Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55f249db53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7182884Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7183122Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55f249db63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7183345Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7183581Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55f249db53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7183810Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7184049Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f249db7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7184342Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7184646Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f249db7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7184869Z frame #60: <unknown function> + 0x147608 (0x55f249dd0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7185108Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f249db7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7185329Z frame #62: <unknown function> + 0x147608 (0x55f249dd0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7185394Z 
2025-07-17T11:07:25.7185492Z SIGABRT(6), PID: 1789277, Thread 1789313: 
2025-07-17T11:07:25.7185984Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7186204Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7186415Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7186623Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7187139Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7187352Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7187569Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7187573Z 
2025-07-17T11:07:25.7187668Z SIGABRT(6), PID: 1789277, Thread 1789314: 
2025-07-17T11:07:25.7188142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7188354Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7188632Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7188836Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7189334Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7189544Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7189756Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7189760Z 
2025-07-17T11:07:25.7189852Z SIGABRT(6), PID: 1789277, Thread 1789316: 
2025-07-17T11:07:25.7190325Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7190534Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7190740Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7190938Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7191434Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7191644Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7191852Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7191857Z 
2025-07-17T11:07:25.7191948Z SIGABRT(6), PID: 1789277, Thread 1789318: 
2025-07-17T11:07:25.7192479Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7192878Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7193091Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7193291Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7193782Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7194059Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7194271Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7194275Z 
2025-07-17T11:07:25.7194368Z SIGABRT(6), PID: 1789277, Thread 1789320: 
2025-07-17T11:07:25.7194840Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7195050Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7195253Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7195456Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7195952Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7196158Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7196367Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7196371Z 
2025-07-17T11:07:25.7196464Z SIGABRT(6), PID: 1789277, Thread 1789321: 
2025-07-17T11:07:25.7197009Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7197215Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7197420Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7197617Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7198121Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7198325Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7198535Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7198540Z 
2025-07-17T11:07:25.7198629Z SIGABRT(6), PID: 1789277, Thread 1789323: 
2025-07-17T11:07:25.7199099Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7199306Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7199511Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7199709Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7200202Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7200409Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7200687Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7200694Z 
2025-07-17T11:07:25.7200844Z SIGABRT(6), PID: 1789277, Thread 1789326: 
2025-07-17T11:07:25.7201318Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7201525Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7201793Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7201992Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7202488Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7202695Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7202912Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7202916Z 
2025-07-17T11:07:25.7203005Z SIGABRT(6), PID: 1789277, Thread 1789329: 
2025-07-17T11:07:25.7203479Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7203685Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7203893Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7204088Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7204589Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7204796Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7205080Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7205085Z 
2025-07-17T11:07:25.7205174Z SIGABRT(6), PID: 1789277, Thread 1789332: 
2025-07-17T11:07:25.7205647Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7205855Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7206059Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7206255Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7206751Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7206962Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7207177Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7207180Z 
2025-07-17T11:07:25.7207270Z SIGABRT(6), PID: 1789277, Thread 1789335: 
2025-07-17T11:07:25.7207738Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7207947Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7208149Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7208347Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7208980Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7209197Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7209408Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7209411Z 
2025-07-17T11:07:25.7209503Z SIGABRT(6), PID: 1789277, Thread 1789338: 
2025-07-17T11:07:25.7210035Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7210243Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7210447Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7210648Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7211145Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7211355Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7211562Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7211565Z 
2025-07-17T11:07:25.7211662Z SIGABRT(6), PID: 1789277, Thread 1789340: 
2025-07-17T11:07:25.7212131Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7212337Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7212540Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7212740Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7213302Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7213512Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7213720Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7213726Z 
2025-07-17T11:07:25.7213816Z SIGABRT(6), PID: 1789277, Thread 1789342: 
2025-07-17T11:07:25.7214280Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7214490Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7214693Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7214894Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7215385Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7215593Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7215803Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7215808Z 
2025-07-17T11:07:25.7215900Z SIGABRT(6), PID: 1789277, Thread 1789345: 
2025-07-17T11:07:25.7216418Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7216710Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7216917Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7217210Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7217710Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7217921Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7218209Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7218218Z 
2025-07-17T11:07:25.7218307Z SIGABRT(6), PID: 1789277, Thread 1789348: 
2025-07-17T11:07:25.7218782Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7218989Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7219195Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7219389Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7219884Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7220091Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7220302Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7220306Z 
2025-07-17T11:07:25.7220393Z SIGABRT(6), PID: 1789277, Thread 1789351: 
2025-07-17T11:07:25.7220864Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7221154Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7221368Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7221563Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7222061Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7222268Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7222480Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7222484Z 
2025-07-17T11:07:25.7222571Z SIGABRT(6), PID: 1789277, Thread 1789353: 
2025-07-17T11:07:25.7223041Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7223247Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7223454Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7223652Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7224154Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7224362Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7224575Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7224579Z 
2025-07-17T11:07:25.7224734Z SIGABRT(6), PID: 1789277, Thread 1789355: 
2025-07-17T11:07:25.7225269Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7225479Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7225686Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7225879Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7226438Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7226645Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7226858Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7226862Z 
2025-07-17T11:07:25.7226951Z SIGABRT(6), PID: 1789277, Thread 1789357: 
2025-07-17T11:07:25.7227426Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7227632Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7227835Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7228031Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7228529Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7228734Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7228948Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7229018Z 
2025-07-17T11:07:25.7229111Z SIGABRT(6), PID: 1789277, Thread 1789360: 
2025-07-17T11:07:25.7229584Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7229791Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7229996Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7230193Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7230687Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7230898Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7231109Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7231114Z 
2025-07-17T11:07:25.7231207Z SIGABRT(6), PID: 1789277, Thread 1789363: 
2025-07-17T11:07:25.7231673Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7231880Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7232086Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7232283Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7232780Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7233058Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7233333Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7233338Z 
2025-07-17T11:07:25.7233432Z SIGABRT(6), PID: 1789277, Thread 1789366: 
2025-07-17T11:07:25.7233898Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7234169Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7234372Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7234571Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7235068Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7235282Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7235488Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7235492Z 
2025-07-17T11:07:25.7235583Z SIGABRT(6), PID: 1789277, Thread 1789368: 
2025-07-17T11:07:25.7236053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7236262Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7236462Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7236659Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7237149Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7237458Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7237666Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7237670Z 
2025-07-17T11:07:25.7237761Z SIGABRT(6), PID: 1789277, Thread 1789370: 
2025-07-17T11:07:25.7238231Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7238439Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7238640Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7238840Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7239333Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7239543Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7239750Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7239760Z 
2025-07-17T11:07:25.7239849Z SIGABRT(6), PID: 1789277, Thread 1789372: 
2025-07-17T11:07:25.7240319Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7240524Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7240730Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7240997Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7241556Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7241766Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7241978Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7242041Z 
2025-07-17T11:07:25.7242132Z SIGABRT(6), PID: 1789277, Thread 1789374: 
2025-07-17T11:07:25.7242600Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7242805Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7243013Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7243211Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7243711Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7243918Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7244131Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7244135Z 
2025-07-17T11:07:25.7244224Z SIGABRT(6), PID: 1789277, Thread 1789377: 
2025-07-17T11:07:25.7244689Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7244894Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7245099Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7245363Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7245861Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7246067Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7246280Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7246283Z 
2025-07-17T11:07:25.7246374Z SIGABRT(6), PID: 1789277, Thread 1789380: 
2025-07-17T11:07:25.7246843Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7247049Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7247259Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7247455Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7247951Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7248157Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7248367Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7248371Z 
2025-07-17T11:07:25.7248458Z SIGABRT(6), PID: 1789277, Thread 1789383: 
2025-07-17T11:07:25.7248995Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7249264Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7249474Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7249670Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7250165Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7250438Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7250648Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7250652Z 
2025-07-17T11:07:25.7250742Z SIGABRT(6), PID: 1789277, Thread 1789386: 
2025-07-17T11:07:25.7251213Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7251430Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7251632Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7251832Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7252328Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7252538Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7252745Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7252749Z 
2025-07-17T11:07:25.7252841Z SIGABRT(6), PID: 1789277, Thread 1789389: 
2025-07-17T11:07:25.7253312Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7253589Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7253792Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7253988Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7254482Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7254692Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7254898Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7254904Z 
2025-07-17T11:07:25.7254995Z SIGABRT(6), PID: 1789277, Thread 1789392: 
2025-07-17T11:07:25.7255464Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7255675Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7255876Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7256130Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7256625Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7256835Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7257125Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7257132Z 
2025-07-17T11:07:25.7257305Z SIGABRT(6), PID: 1789277, Thread 1789394: 
2025-07-17T11:07:25.7257781Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7257991Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7258194Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7258472Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7258971Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7259183Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7259394Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7259402Z 
2025-07-17T11:07:25.7259494Z SIGABRT(6), PID: 1789277, Thread 1789400: 
2025-07-17T11:07:25.7259962Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7260167Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7260369Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7260568Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7261062Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7261272Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7261567Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7261571Z 
2025-07-17T11:07:25.7261660Z SIGABRT(6), PID: 1789277, Thread 1789405: 
2025-07-17T11:07:25.7262132Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7262338Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7262541Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7262738Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7263239Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7263449Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7263662Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7263666Z 
2025-07-17T11:07:25.7263756Z SIGABRT(6), PID: 1789277, Thread 1789408: 
2025-07-17T11:07:25.7264230Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7264436Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7264641Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7264838Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7265401Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7265673Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7265886Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7265890Z 
2025-07-17T11:07:25.7265979Z SIGABRT(6), PID: 1789277, Thread 1789411: 
2025-07-17T11:07:25.7266451Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7266744Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7266953Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7267150Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7267653Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7267863Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7268075Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7268079Z 
2025-07-17T11:07:25.7268168Z SIGABRT(6), PID: 1789277, Thread 1789414: 
2025-07-17T11:07:25.7268640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7268847Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7269051Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7269249Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7269820Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7270029Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7270240Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7270245Z 
2025-07-17T11:07:25.7270335Z SIGABRT(6), PID: 1789277, Thread 1789415: 
2025-07-17T11:07:25.7270810Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7271014Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7271221Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7271417Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7271913Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7272128Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7272342Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7272348Z 
2025-07-17T11:07:25.7272441Z SIGABRT(6), PID: 1789277, Thread 1789417: 
2025-07-17T11:07:25.7272911Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7273119Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7273386Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7273653Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7274152Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7274361Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7274636Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7274640Z 
2025-07-17T11:07:25.7274733Z SIGABRT(6), PID: 1789277, Thread 1789419: 
2025-07-17T11:07:25.7275199Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7275410Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7275615Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7275814Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7276308Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7276519Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7276729Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7276733Z 
2025-07-17T11:07:25.7276825Z SIGABRT(6), PID: 1789277, Thread 1789421: 
2025-07-17T11:07:25.7277289Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7277497Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7277772Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7277973Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7278465Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7278678Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7278885Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7278889Z 
2025-07-17T11:07:25.7278980Z SIGABRT(6), PID: 1789277, Thread 1789423: 
2025-07-17T11:07:25.7279447Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7279658Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7279860Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7280057Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7280551Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7280762Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7280972Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7280976Z 
2025-07-17T11:07:25.7281067Z SIGABRT(6), PID: 1789277, Thread 1789425: 
2025-07-17T11:07:25.7281658Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7281869Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7282077Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7282270Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7282832Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7283038Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7283252Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7283257Z 
2025-07-17T11:07:25.7283346Z SIGABRT(6), PID: 1789277, Thread 1789427: 
2025-07-17T11:07:25.7283819Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7284024Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7284233Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7284431Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7284924Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7285129Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7285342Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7285346Z 
2025-07-17T11:07:25.7285503Z SIGABRT(6), PID: 1789278, Thread 1789278: 
2025-07-17T11:07:25.7285978Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7286186Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7286427Z frame #2: <unknown function> + 0x7741f (0x7fc6f2a2f41f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7286659Z frame #3: <unknown function> + 0x76c8e (0x7fc6f2a2ec8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7286890Z frame #4: <unknown function> + 0x6ad69 (0x7fc6f2a22d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7287105Z frame #5: <unknown function> + 0x3f54db (0x7fc7836874db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7287317Z frame #6: <unknown function> + 0x3e11d8 (0x7fc7836731d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7287529Z frame #7: <unknown function> + 0xb2f86 (0x7fc783344f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7287738Z frame #8: <unknown function> + 0x8e501 (0x7fc783320501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7288127Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fc7ae1277ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.7288516Z frame #10: <unknown function> + 0xd307d9 (0x7fc7cc6767d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7288746Z frame #11: <unknown function> + 0x139902 (0x56373b816902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7288996Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x56373b80da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7289218Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7289527Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x56373b80da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7289814Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7290059Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7290285Z frame #17: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7290522Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7290804Z frame #19: <unknown function> + 0x1474b2 (0x56373b8244b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7291010Z frame #20: PyObject_Call + 0xbc (0x56373b824e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7291232Z frame #21: <unknown function> + 0x1ff69d (0x56373b8dc69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7291457Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x56373b812023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7291698Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x56373b80dcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7291925Z frame #24: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7292160Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x56373b8093bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7292383Z frame #26: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7292616Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x56373b8093bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7292839Z frame #28: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7293075Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x56373b80da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7293333Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x56373b8114b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7293621Z frame #31: _PyObject_Call_Prepend + 0x69 (0x56373b822ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7293844Z frame #32: <unknown function> + 0x205949 (0x56373b8e2949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7294068Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x56373b812023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7294308Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x56373b80dcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7294531Z frame #35: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7294769Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x56373b8093bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7294989Z frame #37: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7295225Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x56373b8093bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7295449Z frame #39: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7295682Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x56373b8093bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7295965Z frame #41: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7296211Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x56373b80a3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7296435Z frame #43: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7296824Z frame #44: <unknown function> + 0x7ab3f7 (0x7fc7cc0f13f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7297044Z frame #45: <unknown function> + 0x13bc88 (0x56373b818c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7297349Z frame #46: PyObject_Call + 0x20f (0x56373b824f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7297689Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x56373b80e98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7297912Z frame #48: <unknown function> + 0x147608 (0x56373b824608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7298152Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x56373b80bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7298376Z frame #50: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7298687Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x56373b8093bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7298915Z frame #52: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7299151Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x56373b80a3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7299377Z frame #54: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7299616Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x56373b8093bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7299839Z frame #56: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7300076Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x56373b80bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7300304Z frame #58: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7300540Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x56373b80bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7300757Z frame #60: <unknown function> + 0x147608 (0x56373b824608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7300993Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x56373b80bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7301212Z frame #62: <unknown function> + 0x147608 (0x56373b824608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7301295Z 
2025-07-17T11:07:25.7301395Z SIGABRT(6), PID: 1789277, Thread 1789429: 
2025-07-17T11:07:25.7301873Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7302087Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7302295Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7302499Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7303001Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7303210Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7303421Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7303428Z 
2025-07-17T11:07:25.7303518Z SIGABRT(6), PID: 1789278, Thread 1789327: 
2025-07-17T11:07:25.7303993Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7304198Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7304407Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7304605Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7305105Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7305387Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7305668Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7305675Z 
2025-07-17T11:07:25.7305770Z SIGABRT(6), PID: 1789277, Thread 1789431: 
2025-07-17T11:07:25.7306241Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7306509Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7306716Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7306913Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7307410Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7307619Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7307831Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7307834Z 
2025-07-17T11:07:25.7307926Z SIGABRT(6), PID: 1789277, Thread 1789433: 
2025-07-17T11:07:25.7308391Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7308600Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7308803Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7309006Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7309498Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7309780Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7309989Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7309993Z 
2025-07-17T11:07:25.7310085Z SIGABRT(6), PID: 1789278, Thread 1789330: 
2025-07-17T11:07:25.7310553Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7310761Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7310963Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7311162Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7311654Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7311865Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7312073Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7312077Z 
2025-07-17T11:07:25.7312167Z SIGABRT(6), PID: 1789280, Thread 1789280: 
2025-07-17T11:07:25.7312632Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7312840Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7313069Z frame #2: <unknown function> + 0x7744b (0x7f2289f6744b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7313369Z frame #3: <unknown function> + 0x76c8e (0x7f2289f66c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7313663Z frame #4: <unknown function> + 0x6ad69 (0x7f2289f5ad69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7313891Z frame #5: <unknown function> + 0x3f54db (0x7f231abbf4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7314104Z frame #6: <unknown function> + 0x3e11d8 (0x7f231abab1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7314315Z frame #7: <unknown function> + 0xb2f86 (0x7f231a87cf86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7314587Z frame #8: <unknown function> + 0x8e501 (0x7f231a858501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7314979Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f234565f7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.7315363Z frame #10: <unknown function> + 0xd307d9 (0x7f2363bae7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7315591Z frame #11: <unknown function> + 0x139902 (0x55ab32f43902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7315843Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55ab32f3aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7316072Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7316312Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55ab32f3aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7316543Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7316779Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7317003Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7317238Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7317462Z frame #19: <unknown function> + 0x1474b2 (0x55ab32f514b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7317735Z frame #20: PyObject_Call + 0xbc (0x55ab32f51e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7317961Z frame #21: <unknown function> + 0x1ff69d (0x55ab3300969d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7318189Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55ab32f3f023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7318433Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55ab32f3acc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7318657Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7318894Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55ab32f363bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7319115Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7319359Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55ab32f363bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7319585Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7319830Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55ab32f3aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7320086Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55ab32f3e4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7320310Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55ab32f4fab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7320527Z frame #32: <unknown function> + 0x205949 (0x55ab3300f949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7320759Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55ab32f3f023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7320997Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55ab32f3acc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7321423Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7321725Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55ab32f363bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7321954Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7322190Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55ab32f363bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7322417Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7322718Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55ab32f363bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7322945Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7323186Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55ab32f373f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7323414Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7323801Z frame #44: <unknown function> + 0x7ab3f7 (0x7f23636293f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7324025Z frame #45: <unknown function> + 0x13bc88 (0x55ab32f45c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7324227Z frame #46: PyObject_Call + 0x20f (0x55ab32f51f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7324476Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55ab32f3b98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7324695Z frame #48: <unknown function> + 0x147608 (0x55ab32f51608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7324937Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55ab32f38c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7325162Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7325400Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55ab32f363bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7325694Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7325935Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55ab32f373f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7326157Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7326395Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55ab32f363bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7326619Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7326857Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55ab32f38c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7327078Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7327320Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55ab32f38c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7327541Z frame #60: <unknown function> + 0x147608 (0x55ab32f51608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7327782Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55ab32f38c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7327998Z frame #62: <unknown function> + 0x147608 (0x55ab32f51608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7328002Z 
2025-07-17T11:07:25.7328098Z SIGABRT(6), PID: 1789277, Thread 1789435: 
2025-07-17T11:07:25.7328568Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7328784Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7328988Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7329284Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7329856Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7330073Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7330284Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7330351Z 
2025-07-17T11:07:25.7330449Z SIGABRT(6), PID: 1789278, Thread 1789334: 
2025-07-17T11:07:25.7330920Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7331135Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7331340Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7331545Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7332044Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7332255Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7332465Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7332469Z 
2025-07-17T11:07:25.7332563Z SIGABRT(6), PID: 1789280, Thread 1789473: 
2025-07-17T11:07:25.7333030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7333238Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7333445Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7333714Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7334211Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7334419Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7334633Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7334636Z 
2025-07-17T11:07:25.7334727Z SIGABRT(6), PID: 1789277, Thread 1789437: 
2025-07-17T11:07:25.7335200Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7335405Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7335611Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7335806Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7336346Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7336555Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7336770Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7336773Z 
2025-07-17T11:07:25.7336863Z SIGABRT(6), PID: 1789278, Thread 1789337: 
2025-07-17T11:07:25.7337423Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7337713Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7337925Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7338120Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7338616Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7338900Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7339109Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7339114Z 
2025-07-17T11:07:25.7339202Z SIGABRT(6), PID: 1789280, Thread 1789474: 
2025-07-17T11:07:25.7339678Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7339888Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7340095Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7340292Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7340786Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7340993Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7341202Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7341207Z 
2025-07-17T11:07:25.7341297Z SIGABRT(6), PID: 1789277, Thread 1789439: 
2025-07-17T11:07:25.7341773Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7342065Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7342272Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7342466Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7342962Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7343169Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7343379Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7343385Z 
2025-07-17T11:07:25.7343478Z SIGABRT(6), PID: 1789278, Thread 1789341: 
2025-07-17T11:07:25.7343956Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7344167Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7344370Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7344573Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7345065Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7345273Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7345542Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7345549Z 
2025-07-17T11:07:25.7345709Z SIGABRT(6), PID: 1789280, Thread 1789475: 
2025-07-17T11:07:25.7346180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7346389Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7346592Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7346858Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7347354Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7347567Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7347774Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7347782Z 
2025-07-17T11:07:25.7347874Z SIGABRT(6), PID: 1789277, Thread 1789441: 
2025-07-17T11:07:25.7348347Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7348554Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7348759Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7348956Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7349449Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7349659Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7349938Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7349942Z 
2025-07-17T11:07:25.7350034Z SIGABRT(6), PID: 1789280, Thread 1789476: 
2025-07-17T11:07:25.7350499Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7350706Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7350908Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7351113Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7351605Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7351816Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7352021Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7352113Z SIGABRT(6), PID: 1789278, Thread 1789343: 
2025-07-17T11:07:25.7352581Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7352790Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7352990Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7353187Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7353743Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7354021Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7354233Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7354238Z 
2025-07-17T11:07:25.7354244Z 
2025-07-17T11:07:25.7354334Z SIGABRT(6), PID: 1789277, Thread 1789443: 
2025-07-17T11:07:25.7354807Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7355074Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7355279Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7355478Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7355978Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7356186Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7356400Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7356404Z 
2025-07-17T11:07:25.7356493Z SIGABRT(6), PID: 1789280, Thread 1789477: 
2025-07-17T11:07:25.7356965Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7357170Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7357374Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7357572Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7358163Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7358369Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7358578Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7358671Z SIGABRT(6), PID: 1789278, Thread 1789346: 
2025-07-17T11:07:25.7359140Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7359344Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7359550Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7359748Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7360245Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7360452Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7360661Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7360667Z 
2025-07-17T11:07:25.7360671Z 
2025-07-17T11:07:25.7360759Z SIGABRT(6), PID: 1789277, Thread 1789445: 
2025-07-17T11:07:25.7361227Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7361496Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7361705Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7361962Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7362461Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7362669Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7362944Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7362947Z 
2025-07-17T11:07:25.7363037Z SIGABRT(6), PID: 1789280, Thread 1789478: 
2025-07-17T11:07:25.7363512Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7363719Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7363931Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7364126Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7364623Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7364830Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7365043Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7365047Z 
2025-07-17T11:07:25.7365137Z SIGABRT(6), PID: 1789278, Thread 1789349: 
2025-07-17T11:07:25.7365611Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7365822Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7366100Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7366300Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7366790Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7367003Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7367210Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7367214Z 
2025-07-17T11:07:25.7367305Z SIGABRT(6), PID: 1789277, Thread 1789447: 
2025-07-17T11:07:25.7367772Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7367985Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7368188Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7368388Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7368878Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7369089Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7369297Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7369301Z 
2025-07-17T11:07:25.7369393Z SIGABRT(6), PID: 1789278, Thread 1789354: 
2025-07-17T11:07:25.7369978Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7370192Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7370397Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7370598Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7371158Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7371375Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7371580Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7371586Z 
2025-07-17T11:07:25.7371681Z SIGABRT(6), PID: 1789280, Thread 1789479: 
2025-07-17T11:07:25.7372153Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7372357Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7372559Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7372759Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7373252Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7373461Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7373672Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7373764Z 
2025-07-17T11:07:25.7373860Z SIGABRT(6), PID: 1789277, Thread 1789449: 
2025-07-17T11:07:25.7374331Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7374537Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7374736Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7374934Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7375427Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7375638Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7375844Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7375853Z 
2025-07-17T11:07:25.7375991Z SIGABRT(6), PID: 1789280, Thread 1789480: 
2025-07-17T11:07:25.7376460Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7376666Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7376873Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7377070Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7377563Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7377858Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7378147Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7378240Z SIGABRT(6), PID: 1789277, Thread 1789451: 
2025-07-17T11:07:25.7378711Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7378918Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7379204Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7379400Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7379902Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7380109Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7380321Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7380325Z 
2025-07-17T11:07:25.7380329Z 
2025-07-17T11:07:25.7380417Z SIGABRT(6), PID: 1789278, Thread 1789356: 
2025-07-17T11:07:25.7380888Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7381094Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7381303Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7381499Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7381992Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7382281Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7382490Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7382494Z 
2025-07-17T11:07:25.7382581Z SIGABRT(6), PID: 1789277, Thread 1789453: 
2025-07-17T11:07:25.7383051Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7383258Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7383463Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7383657Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7384150Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7384360Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7384572Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7384576Z 
2025-07-17T11:07:25.7384665Z SIGABRT(6), PID: 1789280, Thread 1789481: 
2025-07-17T11:07:25.7385134Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7385341Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7385544Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7385808Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7386370Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7386584Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7386791Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7386877Z 
2025-07-17T11:07:25.7386971Z SIGABRT(6), PID: 1789278, Thread 1789358: 
2025-07-17T11:07:25.7387439Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7387646Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7387851Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7388047Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7388543Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7388754Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7388960Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7388965Z 
2025-07-17T11:07:25.7389057Z SIGABRT(6), PID: 1789277, Thread 1789455: 
2025-07-17T11:07:25.7389523Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7389729Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7389932Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7390201Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7390697Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7390904Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7391118Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7391121Z 
2025-07-17T11:07:25.7391216Z SIGABRT(6), PID: 1789280, Thread 1789482: 
2025-07-17T11:07:25.7391683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7391892Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7392097Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7392296Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7392788Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7392999Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7393205Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7393210Z 
2025-07-17T11:07:25.7393300Z SIGABRT(6), PID: 1789278, Thread 1789362: 
2025-07-17T11:07:25.7393828Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7394043Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7394307Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7394508Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7394999Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7395277Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7395483Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7395487Z 
2025-07-17T11:07:25.7395579Z SIGABRT(6), PID: 1789277, Thread 1789457: 
2025-07-17T11:07:25.7396045Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7396257Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7396461Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7396657Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7397149Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7397361Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7397571Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7397575Z 
2025-07-17T11:07:25.7397663Z SIGABRT(6), PID: 1789280, Thread 1789483: 
2025-07-17T11:07:25.7398136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7398411Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7398618Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7398817Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7399312Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7399519Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7399732Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7399736Z 
2025-07-17T11:07:25.7399826Z SIGABRT(6), PID: 1789278, Thread 1789364: 
2025-07-17T11:07:25.7400298Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7400504Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7400711Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7400908Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7401403Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7401609Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7401888Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7401893Z 
2025-07-17T11:07:25.7401985Z SIGABRT(6), PID: 1789277, Thread 1789459: 
2025-07-17T11:07:25.7402515Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7402723Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7402930Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7403189Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7403681Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7403888Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7404102Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7404107Z 
2025-07-17T11:07:25.7404197Z SIGABRT(6), PID: 1789280, Thread 1789484: 
2025-07-17T11:07:25.7404666Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7404870Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7405080Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7405278Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7405776Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7405983Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7406261Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7406266Z 
2025-07-17T11:07:25.7406357Z SIGABRT(6), PID: 1789277, Thread 1789461: 
2025-07-17T11:07:25.7406828Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7407035Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7407240Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7407436Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7407931Z frame #4: <unknown function> + 0x3475ab (0x7f9dc42f05ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7408144Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7408360Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7408364Z 
2025-07-17T11:07:25.7408456Z SIGABRT(6), PID: 1789278, Thread 1789367: 
2025-07-17T11:07:25.7408919Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7409129Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7409332Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7409530Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7410082Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7410359Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7410568Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7410572Z 
2025-07-17T11:07:25.7410666Z SIGABRT(6), PID: 1789280, Thread 1789485: 
2025-07-17T11:07:25.7411127Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7411403Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7411606Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7411804Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7412300Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7412517Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7412722Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7412726Z 
2025-07-17T11:07:25.7412817Z SIGABRT(6), PID: 1789277, Thread 1789810: 
2025-07-17T11:07:25.7413284Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7413742Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f9cd5b4b08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7413947Z frame #2: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7414147Z frame #3: pthread_kill + 0x12c (0x7f9dc745a9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7414382Z frame #4: raise + 0x16 (0x7f9dc7406476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7414545Z frame #5: abort + 0xd3 (0x7f9dc73ec7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7414781Z frame #6: <unknown function> + 0x972cc (0x7f9c436d42cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7415015Z frame #7: <unknown function> + 0x95f47 (0x7f9c436d2f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7415241Z frame #8: <unknown function> + 0x34b71 (0x7f9c43671b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7415451Z frame #9: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7415665Z frame #10: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7415669Z 
2025-07-17T11:07:25.7415763Z SIGABRT(6), PID: 1789280, Thread 1789486: 
2025-07-17T11:07:25.7416278Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7416491Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7416694Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7416893Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7417392Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7417602Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7417811Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7417895Z 
2025-07-17T11:07:25.7417991Z SIGABRT(6), PID: 1789278, Thread 1789373: 
2025-07-17T11:07:25.7418554Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7418766Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7418973Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7419249Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7419744Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7419955Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7420163Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7420260Z SIGABRT(6), PID: 1789277, Thread 1793528: 
2025-07-17T11:07:25.7420730Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7420936Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7421104Z frame #2: __poll + 0x4f (0x7f9dc74dcbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7421516Z frame #3: <unknown function> + 0x489f (0x7f9dc6fe889f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.7421745Z frame #4: <unknown function> + 0x1483bc (0x55f249dd13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7421991Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7422221Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7422544Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7422772Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7423016Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55f249db9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7423248Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7423495Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f249db7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7423724Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7423960Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7424189Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7424426Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7424646Z frame #16: <unknown function> + 0x147734 (0x55f249dd0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7424865Z frame #17: <unknown function> + 0x21ca82 (0x55f249ea5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7425086Z frame #18: <unknown function> + 0x21c9e4 (0x55f249ea59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7425301Z frame #19: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7425517Z frame #20: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7425608Z SIGABRT(6), PID: 1789280, Thread 1789487: 
2025-07-17T11:07:25.7426150Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7426428Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7426642Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7426844Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7427350Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7427625Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7427835Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7427841Z 
2025-07-17T11:07:25.7427844Z 
2025-07-17T11:07:25.7427848Z 
2025-07-17T11:07:25.7427938Z SIGABRT(6), PID: 1789280, Thread 1789488: 
2025-07-17T11:07:25.7428418Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7428628Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7428835Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7429032Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7429530Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7429736Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7429950Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7430040Z SIGABRT(6), PID: 1789278, Thread 1789375: 
2025-07-17T11:07:25.7430582Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7430787Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7430996Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7431192Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7431685Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7431891Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7432103Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7432107Z 
2025-07-17T11:07:25.7432112Z 
2025-07-17T11:07:25.7432200Z SIGABRT(6), PID: 1789280, Thread 1789489: 
2025-07-17T11:07:25.7432666Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7432871Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7433082Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7433278Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7433773Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7434046Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7434261Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7434267Z 
2025-07-17T11:07:25.7434416Z SIGABRT(6), PID: 1789278, Thread 1789378: 
2025-07-17T11:07:25.7434893Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7435103Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7435368Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7435570Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7436059Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7436270Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7436479Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7436483Z 
2025-07-17T11:07:25.7436573Z SIGABRT(6), PID: 1789280, Thread 1789490: 
2025-07-17T11:07:25.7437039Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7437246Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7437451Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7437650Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7438145Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7438353Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7438630Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7438634Z 
2025-07-17T11:07:25.7438727Z SIGABRT(6), PID: 1789277, Thread 1793529: 
2025-07-17T11:07:25.7439195Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7439403Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7439605Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7439834Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f9dc74582dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7440271Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f9d01893048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7440694Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f9d01895bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7440927Z frame #6: <unknown function> + 0xdc253 (0x7f9cd3db7253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7441137Z frame #7: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7441348Z frame #8: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7441352Z 
2025-07-17T11:07:25.7441442Z SIGABRT(6), PID: 1789278, Thread 1789381: 
2025-07-17T11:07:25.7441907Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7442186Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7442452Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7442657Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7443152Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7443426Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7443630Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7443634Z 
2025-07-17T11:07:25.7443730Z SIGABRT(6), PID: 1789280, Thread 1789491: 
2025-07-17T11:07:25.7444193Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7444402Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7444606Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7444807Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7445301Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7445512Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7445717Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7445723Z 
2025-07-17T11:07:25.7445811Z SIGABRT(6), PID: 1789278, Thread 1789384: 
2025-07-17T11:07:25.7446280Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7446665Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7446871Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7447069Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7447564Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7447771Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7447982Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7447986Z 
2025-07-17T11:07:25.7448074Z SIGABRT(6), PID: 1789280, Thread 1789492: 
2025-07-17T11:07:25.7448548Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7448755Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7448959Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7449155Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7449653Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7449858Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7450068Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7450072Z 
2025-07-17T11:07:25.7450247Z SIGABRT(6), PID: 1789277, Thread 1793530: 
2025-07-17T11:07:25.7450780Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7450994Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7451204Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7451426Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f9dc74582dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7451986Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f9d0188d741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7452219Z frame #5: <unknown function> + 0xdc253 (0x7f9cd3db7253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7452433Z frame #6: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7452642Z frame #7: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7452650Z 
2025-07-17T11:07:25.7452741Z SIGABRT(6), PID: 1789277, Thread 1793531: 
2025-07-17T11:07:25.7453210Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7453418Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7453586Z frame #2: ioctl + 0x3f (0x7f9dc74de94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7453824Z frame #3: <unknown function> + 0x13eef0 (0x7f9c4377bef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7454055Z frame #4: <unknown function> + 0x1378c4 (0x7f9c437748c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7454295Z frame #5: <unknown function> + 0x95b2b (0x7f9c436d2b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7454523Z frame #6: <unknown function> + 0x34b71 (0x7f9c43671b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7454804Z frame #7: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7455014Z frame #8: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7455018Z 
2025-07-17T11:07:25.7455110Z SIGABRT(6), PID: 1789278, Thread 1789388: 
2025-07-17T11:07:25.7455575Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7455785Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7456050Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7456256Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7456754Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7456966Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7457173Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7457182Z 
2025-07-17T11:07:25.7457269Z SIGABRT(6), PID: 1789280, Thread 1789493: 
2025-07-17T11:07:25.7457742Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7457945Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7458151Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7458432Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7459006Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7459219Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7459429Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7459504Z 
2025-07-17T11:07:25.7459595Z SIGABRT(6), PID: 1789277, Thread 1793556: 
2025-07-17T11:07:25.7460062Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7460268Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7460442Z frame #2: __poll + 0x4f (0x7f9dc74dcbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7460656Z frame #3: <unknown function> + 0xd4540 (0x7f9c8b8eb540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7460867Z frame #4: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7461076Z frame #5: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7461080Z 
2025-07-17T11:07:25.7461173Z SIGABRT(6), PID: 1789280, Thread 1789494: 
2025-07-17T11:07:25.7461641Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7461849Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7462048Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7462250Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7462746Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7463037Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7463244Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7463337Z SIGABRT(6), PID: 1789278, Thread 1789391: 
2025-07-17T11:07:25.7463806Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7464017Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7464223Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7464428Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7464923Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7465134Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7465339Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7465344Z 
2025-07-17T11:07:25.7465348Z 
2025-07-17T11:07:25.7465439Z SIGABRT(6), PID: 1789277, Thread 1793564: 
2025-07-17T11:07:25.7465906Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7466117Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7466344Z frame #2: __poll + 0x4f (0x7f9dc74dcbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7466623Z frame #3: <unknown function> + 0xd65a2 (0x7f9c8b8ed5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7466834Z frame #4: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7467046Z frame #5: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7467050Z 
2025-07-17T11:07:25.7467138Z SIGABRT(6), PID: 1789277, Thread 1793567: 
2025-07-17T11:07:25.7467671Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7467876Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7468084Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7468281Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7468499Z frame #4: <unknown function> + 0xd1e7f (0x7f9c8b8e8e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7468708Z frame #5: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7468922Z frame #6: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7469013Z SIGABRT(6), PID: 1789278, Thread 1789395: 
2025-07-17T11:07:25.7469477Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7469683Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7469893Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7470090Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7470589Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7470870Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7471078Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7471082Z 
2025-07-17T11:07:25.7471085Z 
2025-07-17T11:07:25.7471177Z SIGABRT(6), PID: 1789280, Thread 1789495: 
2025-07-17T11:07:25.7471644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7471855Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7472059Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7472258Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7472756Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7472966Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7473173Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7473178Z 
2025-07-17T11:07:25.7473268Z SIGABRT(6), PID: 1789280, Thread 1789496: 
2025-07-17T11:07:25.7473734Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7473942Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7474206Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7474468Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7474962Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7475173Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7475443Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7480239Z SIGABRT(6), PID: 1789278, Thread 1789397: 
2025-07-17T11:07:25.7480792Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7481031Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7481250Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7481468Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7481994Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7482211Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7482426Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7482431Z 
2025-07-17T11:07:25.7482435Z 
2025-07-17T11:07:25.7482538Z SIGABRT(6), PID: 1789277, Thread 1793575: 
2025-07-17T11:07:25.7483028Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7483245Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7483599Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7483810Z frame #3: <unknown function> + 0x9ca53 (0x7f9dc7460a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7484070Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55f249d9dbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7484315Z frame #5: <unknown function> + 0x1d314f (0x55f249e5c14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7484541Z frame #6: <unknown function> + 0x1d2f6b (0x55f249e5bf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7484763Z frame #7: <unknown function> + 0x6df36 (0x55f249cf6f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7485008Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7485234Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7485478Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7485711Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7485942Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7486168Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7486401Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7486627Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7486859Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7487153Z frame #17: <unknown function> + 0x147734 (0x55f249dd0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7487447Z frame #18: <unknown function> + 0x21ca82 (0x55f249ea5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7487672Z frame #19: <unknown function> + 0x21c9e4 (0x55f249ea59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7487890Z frame #20: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7488108Z frame #21: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7488173Z 
2025-07-17T11:07:25.7488272Z SIGABRT(6), PID: 1789278, Thread 1789399: 
2025-07-17T11:07:25.7488766Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7488977Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7489188Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7489398Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7489909Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7490115Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7490329Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7490333Z 
2025-07-17T11:07:25.7490423Z SIGABRT(6), PID: 1789280, Thread 1789497: 
2025-07-17T11:07:25.7490894Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7491102Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7491305Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7491578Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7492074Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7492283Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7492494Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7492498Z 
2025-07-17T11:07:25.7492590Z SIGABRT(6), PID: 1789277, Thread 1793585: 
2025-07-17T11:07:25.7493056Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7493269Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7493437Z frame #2: read + 0x4c (0x7f9dc74d881c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7493622Z frame #3: _Py_read + 0x4c (0x55f249e63a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7493849Z frame #4: <unknown function> + 0x208c3b (0x55f249e91c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7494070Z frame #5: <unknown function> + 0x14648d (0x55f249dcf48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7494318Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55f249dde215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7494539Z frame #7: <unknown function> + 0x208ae7 (0x55f249e91ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7494752Z frame #8: <unknown function> + 0x208472 (0x55f249e91472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7495039Z frame #9: <unknown function> + 0x1da6ee (0x55f249e636ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7495270Z frame #10: <unknown function> + 0x1483bc (0x55f249dd13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7495580Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7495808Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7496113Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55f249db53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7496417Z frame #14: <unknown function> + 0x147734 (0x55f249dd0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7496665Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f249db7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7496892Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7497126Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7497349Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55f249dc50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7497583Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55f249db57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7497801Z frame #20: <unknown function> + 0x147734 (0x55f249dd0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7498021Z frame #21: <unknown function> + 0x21ca82 (0x55f249ea5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7498238Z frame #22: <unknown function> + 0x21c9e4 (0x55f249ea59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7498450Z frame #23: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7498661Z frame #24: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7498665Z 
2025-07-17T11:07:25.7498762Z SIGABRT(6), PID: 1789278, Thread 1789402: 
2025-07-17T11:07:25.7499236Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7499536Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7499743Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7499947Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7500445Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7500652Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7500858Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7500862Z 
2025-07-17T11:07:25.7500956Z SIGABRT(6), PID: 1789280, Thread 1789498: 
2025-07-17T11:07:25.7501424Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7501634Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7501838Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7502039Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7502534Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7502741Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7503026Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7503031Z 
2025-07-17T11:07:25.7503131Z SIGABRT(6), PID: 1789278, Thread 1789404: 
2025-07-17T11:07:25.7503663Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7503875Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7504077Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7504343Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7504838Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7505047Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7505255Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7505261Z 
2025-07-17T11:07:25.7505355Z SIGABRT(6), PID: 1789280, Thread 1789499: 
2025-07-17T11:07:25.7505819Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7506027Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7506228Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7506425Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7506914Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7507124Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7507404Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7507409Z 
2025-07-17T11:07:25.7507498Z SIGABRT(6), PID: 1789278, Thread 1789407: 
2025-07-17T11:07:25.7507961Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7508167Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7508372Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7508570Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7509067Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7509271Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7509484Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7509489Z 
2025-07-17T11:07:25.7509576Z SIGABRT(6), PID: 1789280, Thread 1789500: 
2025-07-17T11:07:25.7510039Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7510244Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7510449Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7510642Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7511199Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7511471Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7511682Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7511686Z 
2025-07-17T11:07:25.7511776Z SIGABRT(6), PID: 1789280, Thread 1789501: 
2025-07-17T11:07:25.7512246Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7512535Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7512742Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7512938Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7513435Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7513643Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7513853Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7513857Z 
2025-07-17T11:07:25.7513945Z SIGABRT(6), PID: 1789278, Thread 1789410: 
2025-07-17T11:07:25.7514414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7514617Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7514819Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7515016Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7515512Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7515787Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7515996Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7516000Z 
2025-07-17T11:07:25.7516090Z SIGABRT(6), PID: 1789278, Thread 1789413: 
2025-07-17T11:07:25.7516558Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7516760Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7516964Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7517162Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7517657Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7517865Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7518069Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7518074Z 
2025-07-17T11:07:25.7518166Z SIGABRT(6), PID: 1789280, Thread 1789502: 
2025-07-17T11:07:25.7518626Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7518833Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7519104Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7519367Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7519864Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7520070Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7520341Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7520346Z 
2025-07-17T11:07:25.7520437Z SIGABRT(6), PID: 1789280, Thread 1789504: 
2025-07-17T11:07:25.7520899Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7521106Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7521308Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7521510Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7522007Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7522214Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7522421Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7522513Z SIGABRT(6), PID: 1789278, Thread 1789416: 
2025-07-17T11:07:25.7522975Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7523182Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7523459Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7523662Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7524150Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7524360Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7524564Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7524569Z 
2025-07-17T11:07:25.7524572Z 
2025-07-17T11:07:25.7524663Z SIGABRT(6), PID: 1789277, Thread 1793606: 
2025-07-17T11:07:25.7525126Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7525334Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7525533Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7525731Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7526106Z frame #4: <unknown function> + 0x5a02023 (0x7f9d0b881023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7526675Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9d0b8851ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7527335Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9d0b87c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7527787Z frame #7: <unknown function> + 0x79d655 (0x7f9d1cd68655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7528029Z frame #8: <unknown function> + 0xdc253 (0x7f9cd3db7253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7528240Z frame #9: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7528462Z frame #10: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7528526Z 
2025-07-17T11:07:25.7528617Z SIGABRT(6), PID: 1789278, Thread 1789418: 
2025-07-17T11:07:25.7529088Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7529291Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7529497Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7529699Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7530197Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7530402Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7530611Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7530615Z 
2025-07-17T11:07:25.7530706Z SIGABRT(6), PID: 1789280, Thread 1789505: 
2025-07-17T11:07:25.7531174Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7531381Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7531656Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7531854Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7532349Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7532554Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7532765Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7532769Z 
2025-07-17T11:07:25.7532857Z SIGABRT(6), PID: 1789278, Thread 1789420: 
2025-07-17T11:07:25.7533327Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7533532Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7533741Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7533937Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7534433Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7534639Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7534848Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7534852Z 
2025-07-17T11:07:25.7534939Z SIGABRT(6), PID: 1789280, Thread 1789507: 
2025-07-17T11:07:25.7535472Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7535765Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7536030Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7536228Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7536723Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7537010Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7537219Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7537223Z 
2025-07-17T11:07:25.7537311Z SIGABRT(6), PID: 1789278, Thread 1789422: 
2025-07-17T11:07:25.7537781Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7537989Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7538192Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7538391Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7538883Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7539090Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7539296Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7539299Z 
2025-07-17T11:07:25.7539393Z SIGABRT(6), PID: 1789280, Thread 1789509: 
2025-07-17T11:07:25.7539938Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7540145Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7540345Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7540540Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7541032Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7541239Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7541447Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7541451Z 
2025-07-17T11:07:25.7541544Z SIGABRT(6), PID: 1789278, Thread 1789424: 
2025-07-17T11:07:25.7542007Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7542212Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7542414Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7542617Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7543109Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7543317Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7543603Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7543610Z 
2025-07-17T11:07:25.7543798Z SIGABRT(6), PID: 1789280, Thread 1789511: 
2025-07-17T11:07:25.7544271Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7544479Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7544747Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7544945Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7545440Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7545647Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7545857Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7545861Z 
2025-07-17T11:07:25.7545952Z SIGABRT(6), PID: 1789278, Thread 1789426: 
2025-07-17T11:07:25.7546415Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7546619Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7546822Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7547018Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7547509Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7547717Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7547992Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7547999Z 
2025-07-17T11:07:25.7548088Z SIGABRT(6), PID: 1789280, Thread 1789513: 
2025-07-17T11:07:25.7548557Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7548762Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7548966Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7549161Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7549650Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7549864Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7550076Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7550080Z 
2025-07-17T11:07:25.7550169Z SIGABRT(6), PID: 1789278, Thread 1789428: 
2025-07-17T11:07:25.7550636Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7550842Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7551046Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7551242Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7551862Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7552077Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7552289Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7552293Z 
2025-07-17T11:07:25.7552381Z SIGABRT(6), PID: 1789280, Thread 1789514: 
2025-07-17T11:07:25.7552912Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7553116Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7553321Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7553518Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7554013Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7554221Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7554429Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7554433Z 
2025-07-17T11:07:25.7554522Z SIGABRT(6), PID: 1789277, Thread 1793607: 
2025-07-17T11:07:25.7554988Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7555189Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7555393Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7555590Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7556038Z frame #4: <unknown function> + 0x5a02023 (0x7f9d0b881023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7556602Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9d0b8851ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7557203Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9d0b87c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7557583Z frame #7: <unknown function> + 0x79d655 (0x7f9d1cd68655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7557819Z frame #8: <unknown function> + 0xdc253 (0x7f9cd3db7253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7558023Z frame #9: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7558243Z frame #10: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7558247Z 
2025-07-17T11:07:25.7558337Z SIGABRT(6), PID: 1789278, Thread 1789430: 
2025-07-17T11:07:25.7558801Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7559008Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7559208Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7559406Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7559965Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7560238Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7560449Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7560453Z 
2025-07-17T11:07:25.7560544Z SIGABRT(6), PID: 1789280, Thread 1789516: 
2025-07-17T11:07:25.7561007Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7561279Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7561482Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7561680Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7562177Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7562388Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7562593Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7562597Z 
2025-07-17T11:07:25.7562688Z SIGABRT(6), PID: 1789278, Thread 1789432: 
2025-07-17T11:07:25.7563155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7563361Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7563563Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7563761Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7564325Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7564532Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7564739Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7564744Z 
2025-07-17T11:07:25.7564835Z SIGABRT(6), PID: 1789280, Thread 1789518: 
2025-07-17T11:07:25.7565296Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7565503Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7565703Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7565901Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7566392Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7566600Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7566805Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7566811Z 
2025-07-17T11:07:25.7566901Z SIGABRT(6), PID: 1789278, Thread 1789434: 
2025-07-17T11:07:25.7567362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7567569Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7567836Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7568102Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7568592Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7568800Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7569067Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7569074Z 
2025-07-17T11:07:25.7569162Z SIGABRT(6), PID: 1789280, Thread 1789520: 
2025-07-17T11:07:25.7569623Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7569827Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7570035Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7570228Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7570719Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7570926Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7571134Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7571138Z 
2025-07-17T11:07:25.7571225Z SIGABRT(6), PID: 1789278, Thread 1789436: 
2025-07-17T11:07:25.7571693Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7571897Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7572174Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7572368Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7572857Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7573062Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7573269Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7573273Z 
2025-07-17T11:07:25.7573361Z SIGABRT(6), PID: 1789280, Thread 1789522: 
2025-07-17T11:07:25.7573826Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7574032Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7574236Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7574432Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7574923Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7575128Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7575336Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7575340Z 
2025-07-17T11:07:25.7575427Z SIGABRT(6), PID: 1789278, Thread 1789438: 
2025-07-17T11:07:25.7576102Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7576316Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7576523Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7576718Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7577407Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7577610Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7577818Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7577822Z 
2025-07-17T11:07:25.7577910Z SIGABRT(6), PID: 1789280, Thread 1789524: 
2025-07-17T11:07:25.7578379Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7578583Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7578784Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7578979Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7579474Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7579681Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7579887Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7579891Z 
2025-07-17T11:07:25.7580063Z SIGABRT(6), PID: 1789278, Thread 1789440: 
2025-07-17T11:07:25.7580528Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7580740Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7580940Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7581138Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7581626Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7581831Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7582045Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7582051Z 
2025-07-17T11:07:25.7582143Z SIGABRT(6), PID: 1789280, Thread 1789526: 
2025-07-17T11:07:25.7582602Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7582805Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7583005Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7583201Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7583690Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7583965Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7584250Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7584255Z 
2025-07-17T11:07:25.7584350Z SIGABRT(6), PID: 1789278, Thread 1789442: 
2025-07-17T11:07:25.7584808Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7585017Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7585281Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7585481Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7585966Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7586172Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7586381Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7586387Z 
2025-07-17T11:07:25.7586477Z SIGABRT(6), PID: 1789277, Thread 1793608: 
2025-07-17T11:07:25.7586942Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7587151Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7587351Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7587547Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7587920Z frame #4: <unknown function> + 0x5a02023 (0x7f9d0b881023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7588559Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9d0b8851ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7589154Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9d0b87c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7589537Z frame #7: <unknown function> + 0x79d655 (0x7f9d1cd68655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7589770Z frame #8: <unknown function> + 0xdc253 (0x7f9cd3db7253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7589975Z frame #9: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7590192Z frame #10: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7590198Z 
2025-07-17T11:07:25.7590292Z SIGABRT(6), PID: 1789280, Thread 1789528: 
2025-07-17T11:07:25.7590758Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7590963Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7591166Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7591366Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7591864Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7592136Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7592354Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7592360Z 
2025-07-17T11:07:25.7592512Z SIGABRT(6), PID: 1789278, Thread 1789444: 
2025-07-17T11:07:25.7592982Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7593185Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7593448Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7593643Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7594137Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7594341Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7594554Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7594559Z 
2025-07-17T11:07:25.7594648Z SIGABRT(6), PID: 1789280, Thread 1789530: 
2025-07-17T11:07:25.7595115Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7595318Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7595520Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7595713Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7596208Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7596414Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7596702Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7596706Z 
2025-07-17T11:07:25.7596796Z SIGABRT(6), PID: 1789278, Thread 1789446: 
2025-07-17T11:07:25.7597261Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7597466Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7597673Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7597867Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7598359Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7598567Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7598775Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7598778Z 
2025-07-17T11:07:25.7598865Z SIGABRT(6), PID: 1789280, Thread 1789532: 
2025-07-17T11:07:25.7599332Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7599542Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7599742Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7599938Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7600553Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7600769Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7600976Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7600980Z 
2025-07-17T11:07:25.7601071Z SIGABRT(6), PID: 1789278, Thread 1789448: 
2025-07-17T11:07:25.7601595Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7601806Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7602008Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7602205Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7602696Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7602900Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7603104Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7603110Z 
2025-07-17T11:07:25.7603200Z SIGABRT(6), PID: 1789280, Thread 1789534: 
2025-07-17T11:07:25.7603660Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7603863Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7604063Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7604259Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7604843Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7605051Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7605256Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7605261Z 
2025-07-17T11:07:25.7605351Z SIGABRT(6), PID: 1789278, Thread 1789450: 
2025-07-17T11:07:25.7605811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7606015Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7606215Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7606415Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7606902Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7607111Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7607317Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7607320Z 
2025-07-17T11:07:25.7607410Z SIGABRT(6), PID: 1789280, Thread 1789536: 
2025-07-17T11:07:25.7607868Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7608147Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7608414Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7608615Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7609103Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7609369Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7609577Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7609584Z 
2025-07-17T11:07:25.7609672Z SIGABRT(6), PID: 1789278, Thread 1789452: 
2025-07-17T11:07:25.7610141Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7610346Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7610553Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7610748Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7611236Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7611442Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7611649Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7611653Z 
2025-07-17T11:07:25.7611743Z SIGABRT(6), PID: 1789280, Thread 1789538: 
2025-07-17T11:07:25.7612210Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7612487Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7612691Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7612886Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7613375Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7613582Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7613794Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7613800Z 
2025-07-17T11:07:25.7613888Z SIGABRT(6), PID: 1789278, Thread 1789454: 
2025-07-17T11:07:25.7614365Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7614570Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7614774Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7614968Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7615469Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7615674Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7615883Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7615887Z 
2025-07-17T11:07:25.7616135Z SIGABRT(6), PID: 1789280, Thread 1789540: 
2025-07-17T11:07:25.7616683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7616891Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7617095Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7617289Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7617861Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7618068Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7618281Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7618285Z 
2025-07-17T11:07:25.7618375Z SIGABRT(6), PID: 1789278, Thread 1789456: 
2025-07-17T11:07:25.7618841Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7619045Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7619250Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7619448Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7619942Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7620146Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7620353Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7620436Z 
2025-07-17T11:07:25.7620530Z SIGABRT(6), PID: 1789277, Thread 1793609: 
2025-07-17T11:07:25.7620995Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7621204Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7621406Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7621604Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7621979Z frame #4: <unknown function> + 0x5a02023 (0x7f9d0b881023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7622542Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9d0b8851ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7623134Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9d0b87c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7623511Z frame #7: <unknown function> + 0x79d655 (0x7f9d1cd68655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7623740Z frame #8: <unknown function> + 0xdc253 (0x7f9cd3db7253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7623946Z frame #9: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7624160Z frame #10: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7624164Z 
2025-07-17T11:07:25.7624255Z SIGABRT(6), PID: 1789278, Thread 1789458: 
2025-07-17T11:07:25.7624848Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7625067Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7625273Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7625475Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7626028Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7626237Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7626446Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7626451Z 
2025-07-17T11:07:25.7626545Z SIGABRT(6), PID: 1789280, Thread 1789542: 
2025-07-17T11:07:25.7627014Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7627223Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7627424Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7627624Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7628113Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7628321Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7628529Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7628596Z 
2025-07-17T11:07:25.7628692Z SIGABRT(6), PID: 1789278, Thread 1789460: 
2025-07-17T11:07:25.7629158Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7629366Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7629565Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7629762Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7630256Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7630466Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7630672Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7630680Z 
2025-07-17T11:07:25.7630771Z SIGABRT(6), PID: 1789280, Thread 1789544: 
2025-07-17T11:07:25.7631236Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7631437Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7631643Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7631837Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7632338Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7632610Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7632886Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7632891Z 
2025-07-17T11:07:25.7632983Z SIGABRT(6), PID: 1789278, Thread 1789462: 
2025-07-17T11:07:25.7633454Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7633737Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7633941Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7634137Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7634631Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7634840Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7635048Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7635051Z 
2025-07-17T11:07:25.7635138Z SIGABRT(6), PID: 1789280, Thread 1789546: 
2025-07-17T11:07:25.7635601Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7635806Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7636010Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7636203Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7636695Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7636975Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7637183Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7637187Z 
2025-07-17T11:07:25.7637274Z SIGABRT(6), PID: 1789278, Thread 1789463: 
2025-07-17T11:07:25.7637740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7637943Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7638148Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7638343Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7638834Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7639039Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7639246Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7639250Z 
2025-07-17T11:07:25.7639337Z SIGABRT(6), PID: 1789280, Thread 1789548: 
2025-07-17T11:07:25.7639801Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7640004Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7640206Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7640467Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7641018Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7641227Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7641439Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7641517Z 
2025-07-17T11:07:25.7641608Z SIGABRT(6), PID: 1789278, Thread 1789464: 
2025-07-17T11:07:25.7642070Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7642274Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7642476Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7642672Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7643165Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7643373Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7643579Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7643585Z 
2025-07-17T11:07:25.7643676Z SIGABRT(6), PID: 1789280, Thread 1789550: 
2025-07-17T11:07:25.7644134Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7644339Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7644543Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7644811Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7645300Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7645507Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7645715Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7645719Z 
2025-07-17T11:07:25.7645808Z SIGABRT(6), PID: 1789278, Thread 1789465: 
2025-07-17T11:07:25.7646268Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7646476Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7646681Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7646878Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7647367Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7647576Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7647780Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7647783Z 
2025-07-17T11:07:25.7647874Z SIGABRT(6), PID: 1789280, Thread 1789552: 
2025-07-17T11:07:25.7648397Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7648608Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7648869Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7649074Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7649563Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7649831Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7650038Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7650042Z 
2025-07-17T11:07:25.7650133Z SIGABRT(6), PID: 1789277, Thread 1793610: 
2025-07-17T11:07:25.7650599Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7650809Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7651009Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7651208Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7651575Z frame #4: <unknown function> + 0x5a02023 (0x7f9d0b881023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7652139Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9d0b8851ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7652730Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9d0b87c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7653181Z frame #7: <unknown function> + 0x79d655 (0x7f9d1cd68655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7653413Z frame #8: <unknown function> + 0xdc253 (0x7f9cd3db7253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7653618Z frame #9: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7653837Z frame #10: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7653843Z 
2025-07-17T11:07:25.7653931Z SIGABRT(6), PID: 1789278, Thread 1789466: 
2025-07-17T11:07:25.7654399Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7654603Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7654807Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7655005Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7655501Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7655704Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7655959Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7655963Z 
2025-07-17T11:07:25.7656055Z SIGABRT(6), PID: 1789280, Thread 1789556: 
2025-07-17T11:07:25.7656521Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7656809Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7657092Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7657293Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7657787Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7658066Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7658281Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7658285Z 
2025-07-17T11:07:25.7658374Z SIGABRT(6), PID: 1789278, Thread 1789467: 
2025-07-17T11:07:25.7658842Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7659046Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7659254Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7659447Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7659937Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7660142Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7660351Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7660438Z SIGABRT(6), PID: 1789280, Thread 1789589: 
2025-07-17T11:07:25.7660905Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7661192Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7661396Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7661593Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7662085Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7662291Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7662500Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7662504Z 
2025-07-17T11:07:25.7662508Z 
2025-07-17T11:07:25.7662597Z SIGABRT(6), PID: 1789280, Thread 1789591: 
2025-07-17T11:07:25.7663059Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7663267Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7663467Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7663663Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7664157Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7664366Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7664573Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7664645Z 
2025-07-17T11:07:25.7664740Z SIGABRT(6), PID: 1789278, Thread 1789468: 
2025-07-17T11:07:25.7665288Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7665498Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7665700Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7665965Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7666459Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7666665Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7666871Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7666877Z 
2025-07-17T11:07:25.7666967Z SIGABRT(6), PID: 1789280, Thread 1789593: 
2025-07-17T11:07:25.7667430Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7667637Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7667837Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7668037Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7668527Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7668735Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7668940Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7669014Z 
2025-07-17T11:07:25.7669109Z SIGABRT(6), PID: 1789278, Thread 1789469: 
2025-07-17T11:07:25.7669570Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7669773Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7669976Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7670175Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7670667Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7670875Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7671085Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7671089Z 
2025-07-17T11:07:25.7671179Z SIGABRT(6), PID: 1789280, Thread 1789595: 
2025-07-17T11:07:25.7671645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7671848Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7672049Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7672242Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7672799Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7673068Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7673285Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7673289Z 
2025-07-17T11:07:25.7673377Z SIGABRT(6), PID: 1789278, Thread 1789470: 
2025-07-17T11:07:25.7673841Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7674106Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7674310Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7674502Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7674991Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7675199Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7675407Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7675411Z 
2025-07-17T11:07:25.7675498Z SIGABRT(6), PID: 1789280, Thread 1789597: 
2025-07-17T11:07:25.7675962Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7676169Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7676373Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7676567Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7677057Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7677332Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7677544Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7677548Z 
2025-07-17T11:07:25.7677636Z SIGABRT(6), PID: 1789278, Thread 1789471: 
2025-07-17T11:07:25.7678105Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7678307Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7678511Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7678708Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7679204Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7679408Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7679618Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7679624Z 
2025-07-17T11:07:25.7679713Z SIGABRT(6), PID: 1789280, Thread 1789598: 
2025-07-17T11:07:25.7680180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7680385Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7680652Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7680850Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7681403Z frame #4: <unknown function> + 0x3475ab (0x7f240aba25ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7681613Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7681819Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7681883Z 
2025-07-17T11:07:25.7681973Z SIGABRT(6), PID: 1789277, Thread 1793611: 
2025-07-17T11:07:25.7682439Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7682643Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7682848Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7683047Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7683419Z frame #4: <unknown function> + 0x5a02023 (0x7f9d0b881023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7683973Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9d0b8851ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7684567Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9d0b87c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7684949Z frame #7: <unknown function> + 0x79d655 (0x7f9d1cd68655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7685179Z frame #8: <unknown function> + 0xdc253 (0x7f9cd3db7253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7685458Z frame #9: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7685672Z frame #10: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7685677Z 
2025-07-17T11:07:25.7685769Z SIGABRT(6), PID: 1789278, Thread 1789472: 
2025-07-17T11:07:25.7686232Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7686444Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7686647Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7686847Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7687342Z frame #4: <unknown function> + 0x3475ab (0x7fc8736685ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7690184Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7690418Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7690423Z 
2025-07-17T11:07:25.7690528Z SIGABRT(6), PID: 1789280, Thread 1789802: 
2025-07-17T11:07:25.7691032Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7691492Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f231c3fe08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7691798Z frame #2: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7692014Z frame #3: pthread_kill + 0x12c (0x7f240dd0c9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7692247Z frame #4: raise + 0x16 (0x7f240dcb8476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7692448Z frame #5: abort + 0xd3 (0x7f240dc9e7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7692698Z frame #6: <unknown function> + 0x972cc (0x7f2289f872cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7692931Z frame #7: <unknown function> + 0x95f47 (0x7f2289f85f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7693242Z frame #8: <unknown function> + 0x34b71 (0x7f2289f24b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7693458Z frame #9: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7693678Z frame #10: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7693682Z 
2025-07-17T11:07:25.7693782Z SIGABRT(6), PID: 1789278, Thread 1789798: 
2025-07-17T11:07:25.7694267Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7694727Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fc784ec608a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7694955Z frame #2: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7695156Z frame #3: pthread_kill + 0x12c (0x7fc8767d29fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7695325Z frame #4: raise + 0x16 (0x7fc87677e476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7695482Z frame #5: abort + 0xd3 (0x7fc8767647f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7695724Z frame #6: <unknown function> + 0x972cc (0x7fc6f2a4f2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7696017Z frame #7: <unknown function> + 0x95f47 (0x7fc6f2a4df47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7696253Z frame #8: <unknown function> + 0x34b71 (0x7fc6f29ecb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7696466Z frame #9: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7696684Z frame #10: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7696689Z 
2025-07-17T11:07:25.7696785Z SIGABRT(6), PID: 1789280, Thread 1792321: 
2025-07-17T11:07:25.7697286Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7697501Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7697674Z frame #2: __poll + 0x4f (0x7f240dd8ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7698089Z frame #3: <unknown function> + 0x489f (0x7f240d89a89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.7698333Z frame #4: <unknown function> + 0x1483bc (0x55ab32f523bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7698760Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7698993Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7699232Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7699458Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7699703Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55ab32f3aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7699937Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7700265Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55ab32f38c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7700569Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7700813Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7701041Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7701351Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7701574Z frame #16: <unknown function> + 0x147734 (0x55ab32f51734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7701793Z frame #17: <unknown function> + 0x21ca82 (0x55ab33026a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7702016Z frame #18: <unknown function> + 0x21c9e4 (0x55ab330269e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7702240Z frame #19: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7702457Z frame #20: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7702464Z 
2025-07-17T11:07:25.7702558Z SIGABRT(6), PID: 1789278, Thread 1792124: 
2025-07-17T11:07:25.7703041Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7703260Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7703425Z frame #2: __poll + 0x4f (0x7fc876854bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7703834Z frame #3: <unknown function> + 0x489f (0x7fc87636089f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.7704059Z frame #4: <unknown function> + 0x1483bc (0x56373b8253bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7704301Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7704528Z frame #6: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7704764Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7704984Z frame #8: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7705232Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x56373b80da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7705456Z frame #10: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7705705Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x56373b80bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7705933Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7706169Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7706388Z frame #14: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7706701Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7706918Z frame #16: <unknown function> + 0x147734 (0x56373b824734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7707140Z frame #17: <unknown function> + 0x21ca82 (0x56373b8f9a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7707358Z frame #18: <unknown function> + 0x21c9e4 (0x56373b8f99e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7707574Z frame #19: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7707784Z frame #20: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7707959Z 
2025-07-17T11:07:25.7708060Z SIGABRT(6), PID: 1789280, Thread 1793318: 
2025-07-17T11:07:25.7708593Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7708819Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7709028Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7709315Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f240dd0a2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7709759Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f2348146048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7710181Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f2348148bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7710417Z frame #6: <unknown function> + 0xdc253 (0x7f231a66a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7710631Z frame #7: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7710843Z frame #8: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7710847Z 
2025-07-17T11:07:25.7710942Z SIGABRT(6), PID: 1789277, Thread 1793612: 
2025-07-17T11:07:25.7711414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7711629Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7711833Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7712042Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7712425Z frame #4: <unknown function> + 0x5a02023 (0x7f9d0b881023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7713002Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9d0b8851ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7713599Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9d0b87c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7713990Z frame #7: <unknown function> + 0x79d655 (0x7f9d1cd68655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7714220Z frame #8: <unknown function> + 0xdc253 (0x7f9cd3db7253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7714431Z frame #9: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7714647Z frame #10: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7714653Z 
2025-07-17T11:07:25.7714747Z SIGABRT(6), PID: 1789280, Thread 1793319: 
2025-07-17T11:07:25.7715297Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7715513Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7715719Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7715947Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f240dd0a2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7716418Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f2348140741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7716709Z frame #5: <unknown function> + 0xdc253 (0x7f231a66a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7716977Z frame #6: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7717199Z frame #7: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7717203Z 
2025-07-17T11:07:25.7717294Z SIGABRT(6), PID: 1789280, Thread 1793321: 
2025-07-17T11:07:25.7717775Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7718047Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7718216Z frame #2: ioctl + 0x3f (0x7f240dd9094f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7718456Z frame #3: <unknown function> + 0x13eef0 (0x7f228a02eef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7718690Z frame #4: <unknown function> + 0x1378c4 (0x7f228a0278c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7718923Z frame #5: <unknown function> + 0x95b2b (0x7f2289f85b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7719154Z frame #6: <unknown function> + 0x34b71 (0x7f2289f24b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7719363Z frame #7: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7719578Z frame #8: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7719584Z 
2025-07-17T11:07:25.7719677Z SIGABRT(6), PID: 1789278, Thread 1793535: 
2025-07-17T11:07:25.7720152Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7720362Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7720568Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7720798Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fc8767d02dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7721244Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fc7b0c0e048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7721667Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fc7b0c10bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7721898Z frame #6: <unknown function> + 0xdc253 (0x7fc783132253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7722106Z frame #7: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7722314Z frame #8: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7722318Z 
2025-07-17T11:07:25.7722413Z SIGABRT(6), PID: 1789280, Thread 1793550: 
2025-07-17T11:07:25.7722894Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7723173Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7723343Z frame #2: __poll + 0x4f (0x7f240dd8ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7723559Z frame #3: <unknown function> + 0xd4540 (0x7f22d219e540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7723767Z frame #4: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7723980Z frame #5: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7723984Z 
2025-07-17T11:07:25.7724074Z SIGABRT(6), PID: 1789278, Thread 1793536: 
2025-07-17T11:07:25.7724610Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7724874Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7725091Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7725317Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fc8767d02dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7725791Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fc7b0c08741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7726103Z frame #5: <unknown function> + 0xdc253 (0x7fc783132253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7726314Z frame #6: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7726525Z frame #7: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7726529Z 
2025-07-17T11:07:25.7726622Z SIGABRT(6), PID: 1789280, Thread 1793558: 
2025-07-17T11:07:25.7727091Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7727305Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7727470Z frame #2: __poll + 0x4f (0x7f240dd8ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7727691Z frame #3: <unknown function> + 0xd65a2 (0x7f22d21a05a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7727899Z frame #4: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7728109Z frame #5: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7728113Z 
2025-07-17T11:07:25.7728202Z SIGABRT(6), PID: 1789278, Thread 1793539: 
2025-07-17T11:07:25.7728674Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7728882Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7729051Z frame #2: ioctl + 0x3f (0x7fc87685694f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7729287Z frame #3: <unknown function> + 0x13eef0 (0x7fc6f2af6ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7729524Z frame #4: <unknown function> + 0x1378c4 (0x7fc6f2aef8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7729755Z frame #5: <unknown function> + 0x95b2b (0x7fc6f2a4db2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7729984Z frame #6: <unknown function> + 0x34b71 (0x7fc6f29ecb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7730194Z frame #7: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7730411Z frame #8: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7730415Z 
2025-07-17T11:07:25.7730505Z SIGABRT(6), PID: 1789280, Thread 1793568: 
2025-07-17T11:07:25.7730981Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7731255Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7731466Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7731668Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7731888Z frame #4: <unknown function> + 0xd1e7f (0x7f22d219be7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7732092Z frame #5: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7732376Z frame #6: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7732381Z 
2025-07-17T11:07:25.7732528Z SIGABRT(6), PID: 1789278, Thread 1793555: 
2025-07-17T11:07:25.7733008Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7733219Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7733441Z frame #2: __poll + 0x4f (0x7fc876854bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7733660Z frame #3: <unknown function> + 0xd4540 (0x7fc73ac66540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7733866Z frame #4: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7734072Z frame #5: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7734080Z 
2025-07-17T11:07:25.7734171Z SIGABRT(6), PID: 1789280, Thread 1793574: 
2025-07-17T11:07:25.7734644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7734853Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7735060Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7735266Z frame #3: <unknown function> + 0x9ca53 (0x7f240dd12a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7735527Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55ab32f1ebc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7735756Z frame #5: <unknown function> + 0x1d314f (0x55ab32fdd14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7736032Z frame #6: <unknown function> + 0x1d2f6b (0x55ab32fdcf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7736258Z frame #7: <unknown function> + 0x6df36 (0x55ab32e77f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7736506Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7736732Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7736977Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7737204Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7737446Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7737671Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7737909Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7738133Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7738373Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7738594Z frame #17: <unknown function> + 0x147734 (0x55ab32f51734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7738897Z frame #18: <unknown function> + 0x21ca82 (0x55ab33026a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7739116Z frame #19: <unknown function> + 0x21c9e4 (0x55ab330269e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7739334Z frame #20: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7739546Z frame #21: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7739550Z 
2025-07-17T11:07:25.7739645Z SIGABRT(6), PID: 1789277, Thread 1793613: 
2025-07-17T11:07:25.7740200Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9cd5b4ab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7740485Z frame #1: <unknown function> + 0x42520 (0x7f9dc7406520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7740699Z frame #2: <unknown function> + 0x91117 (0x7f9dc7455117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7740906Z frame #3: pthread_cond_wait + 0x211 (0x7f9dc7457a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7741284Z frame #4: <unknown function> + 0x5a02023 (0x7f9d0b881023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7741928Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9d0b8851ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7742526Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9d0b87c1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7742916Z frame #7: <unknown function> + 0x79d655 (0x7f9d1cd68655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7743151Z frame #8: <unknown function> + 0xdc253 (0x7f9cd3db7253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7743365Z frame #9: <unknown function> + 0x94ac3 (0x7f9dc7458ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7743582Z frame #10: <unknown function> + 0x126850 (0x7f9dc74ea850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7743588Z 
2025-07-17T11:07:25.7743684Z SIGABRT(6), PID: 1789278, Thread 1793563: 
2025-07-17T11:07:25.7744157Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7744370Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7744531Z frame #2: __poll + 0x4f (0x7fc876854bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7744752Z frame #3: <unknown function> + 0xd65a2 (0x7fc73ac685a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7744960Z frame #4: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7745170Z frame #5: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7745173Z 
2025-07-17T11:07:25.7745266Z SIGABRT(6), PID: 1789280, Thread 1793583: 
2025-07-17T11:07:25.7745737Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7745946Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7746114Z frame #2: read + 0x4c (0x7f240dd8a81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7746300Z frame #3: _Py_read + 0x4c (0x55ab32fe4a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7746531Z frame #4: <unknown function> + 0x208c3b (0x55ab33012c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7746752Z frame #5: <unknown function> + 0x14648d (0x55ab32f5048d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7747070Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55ab32f5f215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7747292Z frame #7: <unknown function> + 0x208ae7 (0x55ab33012ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7747515Z frame #8: <unknown function> + 0x208472 (0x55ab33012472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7747732Z frame #9: <unknown function> + 0x1da6ee (0x55ab32fe46ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7747963Z frame #10: <unknown function> + 0x1483bc (0x55ab32f523bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7748263Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7748555Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7748798Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55ab32f363bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7749026Z frame #14: <unknown function> + 0x147734 (0x55ab32f51734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7749275Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55ab32f38c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7749567Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7749803Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7750027Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55ab32f460ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7750262Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55ab32f367b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7750484Z frame #20: <unknown function> + 0x147734 (0x55ab32f51734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7750701Z frame #21: <unknown function> + 0x21ca82 (0x55ab33026a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7750923Z frame #22: <unknown function> + 0x21c9e4 (0x55ab330269e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7751136Z frame #23: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7751353Z frame #24: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7751358Z 
2025-07-17T11:07:25.7751448Z SIGABRT(6), PID: 1789278, Thread 1793571: 
2025-07-17T11:07:25.7751927Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7752138Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7752353Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7752554Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7752776Z frame #4: <unknown function> + 0xd1e7f (0x7fc73ac63e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.7752984Z frame #5: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7753196Z frame #6: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7753200Z 
2025-07-17T11:07:25.7753291Z SIGABRT(6), PID: 1789278, Thread 1793581: 
2025-07-17T11:07:25.7753761Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7753970Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7754178Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7754386Z frame #3: <unknown function> + 0x9ca53 (0x7fc8767d8a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7754709Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x56373b7f1bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7754931Z frame #5: <unknown function> + 0x1d314f (0x56373b8b014f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7755160Z frame #6: <unknown function> + 0x1d2f6b (0x56373b8aff6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7755384Z frame #7: <unknown function> + 0x6df36 (0x56373b74af36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7755625Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7755909Z frame #9: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7756152Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7756456Z frame #11: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7756694Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7756920Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7757214Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7757439Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7757673Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7757896Z frame #17: <unknown function> + 0x147734 (0x56373b824734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7758115Z frame #18: <unknown function> + 0x21ca82 (0x56373b8f9a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7758341Z frame #19: <unknown function> + 0x21c9e4 (0x56373b8f99e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7758552Z frame #20: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7758763Z frame #21: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7758769Z 
2025-07-17T11:07:25.7758859Z SIGABRT(6), PID: 1789278, Thread 1793596: 
2025-07-17T11:07:25.7759332Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7759542Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7759709Z frame #2: read + 0x4c (0x7fc87685081c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7759886Z frame #3: _Py_read + 0x4c (0x56373b8b7a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7760113Z frame #4: <unknown function> + 0x208c3b (0x56373b8e5c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7760334Z frame #5: <unknown function> + 0x14648d (0x56373b82348d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7760578Z frame #6: PyObject_VectorcallMethod + 0x85 (0x56373b832215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7760799Z frame #7: <unknown function> + 0x208ae7 (0x56373b8e5ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7761020Z frame #8: <unknown function> + 0x208472 (0x56373b8e5472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7761240Z frame #9: <unknown function> + 0x1da6ee (0x56373b8b76ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7761464Z frame #10: <unknown function> + 0x1483bc (0x56373b8253bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7761709Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7761938Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7762174Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x56373b8093bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7762462Z frame #14: <unknown function> + 0x147734 (0x56373b824734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7762714Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x56373b80bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7762942Z frame #16: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7763175Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7763398Z frame #18: _PyFunction_Vectorcall + 0x6c (0x56373b8190ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7763689Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x56373b8097b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7763965Z frame #20: <unknown function> + 0x147734 (0x56373b824734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7764187Z frame #21: <unknown function> + 0x21ca82 (0x56373b8f9a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7764409Z frame #22: <unknown function> + 0x21c9e4 (0x56373b8f99e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7764619Z frame #23: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7764892Z frame #24: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7764898Z 
2025-07-17T11:07:25.7764989Z SIGABRT(6), PID: 1789280, Thread 1793598: 
2025-07-17T11:07:25.7765464Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7765675Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7765883Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7766087Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7766468Z frame #4: <unknown function> + 0x5a02023 (0x7f2352134023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7767032Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f23521381ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7767629Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f235212f1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7768015Z frame #7: <unknown function> + 0x79d655 (0x7f236361b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7768249Z frame #8: <unknown function> + 0xdc253 (0x7f231a66a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7768459Z frame #9: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7768676Z frame #10: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7768680Z 
2025-07-17T11:07:25.7768774Z SIGABRT(6), PID: 1789278, Thread 1793646: 
2025-07-17T11:07:25.7769245Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7769455Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7769660Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7769867Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7770249Z frame #4: <unknown function> + 0x5a02023 (0x7fc7babfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7770891Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc7bac001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7771487Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc7babf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7771878Z frame #7: <unknown function> + 0x79d655 (0x7fc7cc0e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7772162Z frame #8: <unknown function> + 0xdc253 (0x7fc783132253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7772375Z frame #9: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7772659Z frame #10: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7772666Z 
2025-07-17T11:07:25.7772762Z SIGABRT(6), PID: 1789280, Thread 1793599: 
2025-07-17T11:07:25.7773241Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7773511Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7773717Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7773922Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7774291Z frame #4: <unknown function> + 0x5a02023 (0x7f2352134023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7774859Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f23521381ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7775452Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f235212f1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7775837Z frame #7: <unknown function> + 0x79d655 (0x7f236361b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7776125Z frame #8: <unknown function> + 0xdc253 (0x7f231a66a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7776339Z frame #9: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7776555Z frame #10: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7776559Z 
2025-07-17T11:07:25.7776655Z SIGABRT(6), PID: 1789278, Thread 1793647: 
2025-07-17T11:07:25.7777131Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7777341Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7777545Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7777749Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7778120Z frame #4: <unknown function> + 0x5a02023 (0x7fc7babfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7778684Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc7bac001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7779277Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc7babf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7779741Z frame #7: <unknown function> + 0x79d655 (0x7fc7cc0e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7779975Z frame #8: <unknown function> + 0xdc253 (0x7fc783132253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7780187Z frame #9: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7780400Z frame #10: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7780404Z 
2025-07-17T11:07:25.7780494Z SIGABRT(6), PID: 1789280, Thread 1793600: 
2025-07-17T11:07:25.7781043Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7781332Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7781562Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7781772Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7782225Z frame #4: <unknown function> + 0x5a02023 (0x7f2352134023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7782795Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f23521381ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7783391Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f235212f1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7783776Z frame #7: <unknown function> + 0x79d655 (0x7f236361b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7784011Z frame #8: <unknown function> + 0xdc253 (0x7f231a66a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7784219Z frame #9: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7784436Z frame #10: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7784440Z 
2025-07-17T11:07:25.7784529Z SIGABRT(6), PID: 1789278, Thread 1793648: 
2025-07-17T11:07:25.7785007Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7785214Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7785425Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7785623Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7785999Z frame #4: <unknown function> + 0x5a02023 (0x7fc7babfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7786561Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc7bac001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7787157Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc7babf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7787544Z frame #7: <unknown function> + 0x79d655 (0x7fc7cc0e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7787772Z frame #8: <unknown function> + 0xdc253 (0x7fc783132253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7788053Z frame #9: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7788266Z frame #10: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7788270Z 
2025-07-17T11:07:25.7788365Z SIGABRT(6), PID: 1789280, Thread 1793601: 
2025-07-17T11:07:25.7788836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7789045Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7789246Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7789526Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7789955Z frame #4: <unknown function> + 0x5a02023 (0x7f2352134023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7790522Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f23521381ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7791171Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f235212f1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7791553Z frame #7: <unknown function> + 0x79d655 (0x7f236361b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7791783Z frame #8: <unknown function> + 0xdc253 (0x7f231a66a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7791994Z frame #9: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7792208Z frame #10: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7792214Z 
2025-07-17T11:07:25.7792306Z SIGABRT(6), PID: 1789278, Thread 1793649: 
2025-07-17T11:07:25.7792777Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7792986Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7793190Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7793392Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7793762Z frame #4: <unknown function> + 0x5a02023 (0x7fc7babfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7794331Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc7bac001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7794923Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc7babf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7795309Z frame #7: <unknown function> + 0x79d655 (0x7fc7cc0e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7795537Z frame #8: <unknown function> + 0xdc253 (0x7fc783132253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7795746Z frame #9: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7795962Z frame #10: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7795969Z 
2025-07-17T11:07:25.7796060Z SIGABRT(6), PID: 1789280, Thread 1793602: 
2025-07-17T11:07:25.7796534Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7796812Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7797016Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7797218Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7797589Z frame #4: <unknown function> + 0x5a02023 (0x7f2352134023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7798203Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f23521381ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7798855Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f235212f1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7799236Z frame #7: <unknown function> + 0x79d655 (0x7f236361b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7799530Z frame #8: <unknown function> + 0xdc253 (0x7f231a66a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7799739Z frame #9: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7799955Z frame #10: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7799959Z 
2025-07-17T11:07:25.7800048Z SIGABRT(6), PID: 1789278, Thread 1793650: 
2025-07-17T11:07:25.7800531Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7800741Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7800952Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7801151Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7801531Z frame #4: <unknown function> + 0x5a02023 (0x7fc7babfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7802087Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc7bac001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7802681Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc7babf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7803059Z frame #7: <unknown function> + 0x79d655 (0x7fc7cc0e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7803289Z frame #8: <unknown function> + 0xdc253 (0x7fc783132253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7803496Z frame #9: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7803711Z frame #10: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7803715Z 
2025-07-17T11:07:25.7803803Z SIGABRT(6), PID: 1789280, Thread 1793603: 
2025-07-17T11:07:25.7804277Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7804485Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7804693Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7804891Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7805331Z frame #4: <unknown function> + 0x5a02023 (0x7f2352134023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7805889Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f23521381ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7806484Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f235212f1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7806921Z frame #7: <unknown function> + 0x79d655 (0x7f236361b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7807210Z frame #8: <unknown function> + 0xdc253 (0x7f231a66a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7807431Z frame #9: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7807646Z frame #10: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7807650Z 
2025-07-17T11:07:25.7807804Z SIGABRT(6), PID: 1789278, Thread 1793651: 
2025-07-17T11:07:25.7808281Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7808490Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7808695Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7808897Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7809269Z frame #4: <unknown function> + 0x5a02023 (0x7fc7babfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7809832Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc7bac001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7810420Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc7babf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7810803Z frame #7: <unknown function> + 0x79d655 (0x7fc7cc0e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7811030Z frame #8: <unknown function> + 0xdc253 (0x7fc783132253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7811241Z frame #9: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7811456Z frame #10: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7811462Z 
2025-07-17T11:07:25.7811554Z SIGABRT(6), PID: 1789280, Thread 1793604: 
2025-07-17T11:07:25.7812024Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7812233Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7812436Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7812638Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7813007Z frame #4: <unknown function> + 0x5a02023 (0x7f2352134023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7813564Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f23521381ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7814219Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f235212f1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7814601Z frame #7: <unknown function> + 0x79d655 (0x7f236361b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7814831Z frame #8: <unknown function> + 0xdc253 (0x7f231a66a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7815038Z frame #9: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7815309Z frame #10: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7815313Z 
2025-07-17T11:07:25.7815404Z SIGABRT(6), PID: 1789278, Thread 1793652: 
2025-07-17T11:07:25.7815987Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7816203Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7816409Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7816685Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7817060Z frame #4: <unknown function> + 0x5a02023 (0x7fc7babfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7817621Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc7bac001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7818209Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc7babf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7818592Z frame #7: <unknown function> + 0x79d655 (0x7fc7cc0e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7818823Z frame #8: <unknown function> + 0xdc253 (0x7fc783132253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7819031Z frame #9: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7819245Z frame #10: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7819249Z 
2025-07-17T11:07:25.7819337Z SIGABRT(6), PID: 1789280, Thread 1793605: 
2025-07-17T11:07:25.7819813Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f231c3fdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7820022Z frame #1: <unknown function> + 0x42520 (0x7f240dcb8520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7820230Z frame #2: <unknown function> + 0x91117 (0x7f240dd07117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7820429Z frame #3: pthread_cond_wait + 0x211 (0x7f240dd09a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7820800Z frame #4: <unknown function> + 0x5a02023 (0x7f2352134023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7821355Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f23521381ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7821941Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f235212f1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7822321Z frame #7: <unknown function> + 0x79d655 (0x7f236361b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7822662Z frame #8: <unknown function> + 0xdc253 (0x7f231a66a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7822872Z frame #9: <unknown function> + 0x94ac3 (0x7f240dd0aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7823091Z frame #10: <unknown function> + 0x126850 (0x7f240dd9c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7823095Z 
2025-07-17T11:07:25.7823184Z SIGABRT(6), PID: 1789278, Thread 1793653: 
2025-07-17T11:07:25.7823657Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc784ec5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7823941Z frame #1: <unknown function> + 0x42520 (0x7fc87677e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7824216Z frame #2: <unknown function> + 0x91117 (0x7fc8767cd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7824423Z frame #3: pthread_cond_wait + 0x211 (0x7fc8767cfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7824794Z frame #4: <unknown function> + 0x5a02023 (0x7fc7babfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7825355Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc7bac001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7826009Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc7babf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.7826392Z frame #7: <unknown function> + 0x79d655 (0x7fc7cc0e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7826620Z frame #8: <unknown function> + 0xdc253 (0x7fc783132253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7826829Z frame #9: <unknown function> + 0x94ac3 (0x7fc8767d0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7827040Z frame #10: <unknown function> + 0x126850 (0x7fc876862850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7827044Z 
2025-07-17T11:07:25.7827136Z SIGABRT(6), PID: 1789279, Thread 1789279: 
2025-07-17T11:07:25.7827610Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7827819Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7828051Z frame #2: <unknown function> + 0x77426 (0x7fa86c7fe426 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7828285Z frame #3: <unknown function> + 0x76c8e (0x7fa86c7fdc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7828511Z frame #4: <unknown function> + 0x6ad69 (0x7fa86c7f1d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7828731Z frame #5: <unknown function> + 0x3f54db (0x7fa8fd4564db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7828942Z frame #6: <unknown function> + 0x3e11d8 (0x7fa8fd4421d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7829155Z frame #7: <unknown function> + 0xb2f86 (0x7fa8fd113f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7829365Z frame #8: <unknown function> + 0x8e501 (0x7fa8fd0ef501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.7829746Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fa927ef67ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.7830127Z frame #10: <unknown function> + 0xd307d9 (0x7fa9464497d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7830355Z frame #11: <unknown function> + 0x139902 (0x56424bc1c902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7830609Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x56424bc13a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7830907Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7831153Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x56424bc13a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7831384Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7831624Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7831849Z frame #17: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7832149Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7832377Z frame #19: <unknown function> + 0x1474b2 (0x56424bc2a4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7832643Z frame #20: PyObject_Call + 0xbc (0x56424bc2ae4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7832870Z frame #21: <unknown function> + 0x1ff69d (0x56424bce269d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7833093Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x56424bc18023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7833400Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x56424bc13cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7833623Z frame #24: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7833862Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x56424bc0f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7834082Z frame #26: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7834320Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x56424bc0f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7834544Z frame #28: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7834788Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x56424bc13a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7835046Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x56424bc174b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7835271Z frame #31: _PyObject_Call_Prepend + 0x69 (0x56424bc28ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7835493Z frame #32: <unknown function> + 0x205949 (0x56424bce8949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7835718Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x56424bc18023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7835961Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x56424bc13cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7836191Z frame #35: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7836428Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x56424bc0f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7836654Z frame #37: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7836889Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x56424bc0f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7837114Z frame #39: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7837348Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x56424bc0f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7837575Z frame #41: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7837815Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x56424bc103f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7838040Z frame #43: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7838428Z frame #44: <unknown function> + 0x7ab3f7 (0x7fa945ec43f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.7838841Z frame #45: <unknown function> + 0x13bc88 (0x56424bc1ec88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7839047Z frame #46: PyObject_Call + 0x20f (0x56424bc2af9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7839293Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x56424bc1498d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7839514Z frame #48: <unknown function> + 0x147608 (0x56424bc2a608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7839756Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x56424bc11c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7839978Z frame #50: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7840275Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x56424bc0f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7840554Z frame #52: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7840801Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x56424bc103f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7841022Z frame #54: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7841324Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x56424bc0f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7841545Z frame #56: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7841783Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x56424bc11c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7842005Z frame #58: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7842245Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x56424bc11c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7842463Z frame #60: <unknown function> + 0x147608 (0x56424bc2a608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7842707Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x56424bc11c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7842922Z frame #62: <unknown function> + 0x147608 (0x56424bc2a608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7842926Z 
2025-07-17T11:07:25.7843025Z SIGABRT(6), PID: 1789279, Thread 1789503: 
2025-07-17T11:07:25.7843502Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7843716Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7843929Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7844133Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7844643Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7844856Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7845069Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7845075Z 
2025-07-17T11:07:25.7845172Z SIGABRT(6), PID: 1789279, Thread 1789506: 
2025-07-17T11:07:25.7845644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7845851Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7846056Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7846260Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7846758Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7847043Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7847259Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7847264Z 
2025-07-17T11:07:25.7847359Z SIGABRT(6), PID: 1789279, Thread 1789508: 
2025-07-17T11:07:25.7847826Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7848093Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7848300Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7848556Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7849056Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7849267Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7849563Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7849569Z 
2025-07-17T11:07:25.7849660Z SIGABRT(6), PID: 1789279, Thread 1789510: 
2025-07-17T11:07:25.7850132Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7850340Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7850545Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7850743Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7851242Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7851451Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7851662Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7851666Z 
2025-07-17T11:07:25.7851754Z SIGABRT(6), PID: 1789279, Thread 1789512: 
2025-07-17T11:07:25.7852228Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7852435Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7852640Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7852837Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7853329Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7853539Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7859048Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7859059Z 
2025-07-17T11:07:25.7859197Z SIGABRT(6), PID: 1789279, Thread 1789515: 
2025-07-17T11:07:25.7859730Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7859972Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7860341Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7860565Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7861101Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7861320Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7861549Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7861553Z 
2025-07-17T11:07:25.7861754Z SIGABRT(6), PID: 1789279, Thread 1789517: 
2025-07-17T11:07:25.7862338Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7862562Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7862772Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7862980Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7863575Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7863786Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7864002Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7864007Z 
2025-07-17T11:07:25.7864101Z SIGABRT(6), PID: 1789279, Thread 1789519: 
2025-07-17T11:07:25.7864579Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7864790Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7864996Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7865197Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7865706Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7865914Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7866127Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7866132Z 
2025-07-17T11:07:25.7866223Z SIGABRT(6), PID: 1789279, Thread 1789521: 
2025-07-17T11:07:25.7866697Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7866906Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7867109Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7867310Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7867803Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7868013Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7868224Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7868231Z 
2025-07-17T11:07:25.7868321Z SIGABRT(6), PID: 1789279, Thread 1789523: 
2025-07-17T11:07:25.7868861Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7869067Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7869272Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7869471Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7869960Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7870230Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7870497Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7870504Z 
2025-07-17T11:07:25.7870600Z SIGABRT(6), PID: 1789279, Thread 1789525: 
2025-07-17T11:07:25.7871066Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7871332Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7871535Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7871752Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7872272Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7872491Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7872700Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7872707Z 
2025-07-17T11:07:25.7872801Z SIGABRT(6), PID: 1789279, Thread 1789527: 
2025-07-17T11:07:25.7873278Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7873491Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7873693Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7873897Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7874409Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7874623Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7874833Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7874837Z 
2025-07-17T11:07:25.7874930Z SIGABRT(6), PID: 1789279, Thread 1789529: 
2025-07-17T11:07:25.7875407Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7875618Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7875822Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7876028Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7876525Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7876810Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7877019Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7877026Z 
2025-07-17T11:07:25.7877119Z SIGABRT(6), PID: 1789279, Thread 1789531: 
2025-07-17T11:07:25.7877589Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7877793Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7877998Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7878257Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7878815Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7879029Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7879239Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7879300Z 
2025-07-17T11:07:25.7879393Z SIGABRT(6), PID: 1789279, Thread 1789533: 
2025-07-17T11:07:25.7879868Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7880074Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7880282Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7880483Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7880982Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7881191Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7881403Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7881407Z 
2025-07-17T11:07:25.7881497Z SIGABRT(6), PID: 1789279, Thread 1789535: 
2025-07-17T11:07:25.7881963Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7882169Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7882377Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7882583Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7883078Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7883287Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7883497Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7883501Z 
2025-07-17T11:07:25.7883591Z SIGABRT(6), PID: 1789279, Thread 1789537: 
2025-07-17T11:07:25.7884064Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7884271Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7884476Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7884672Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7885242Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7885451Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7885658Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7885663Z 
2025-07-17T11:07:25.7885750Z SIGABRT(6), PID: 1789279, Thread 1789539: 
2025-07-17T11:07:25.7886282Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7886569Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7886778Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7886978Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7887474Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7887742Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7887951Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7887955Z 
2025-07-17T11:07:25.7888047Z SIGABRT(6), PID: 1789279, Thread 1789541: 
2025-07-17T11:07:25.7888512Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7888723Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7888927Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7889125Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7889616Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7889826Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7890033Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7890037Z 
2025-07-17T11:07:25.7890129Z SIGABRT(6), PID: 1789279, Thread 1789543: 
2025-07-17T11:07:25.7890597Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7890803Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7891006Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7891204Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7891699Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7891908Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7892122Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7892128Z 
2025-07-17T11:07:25.7892220Z SIGABRT(6), PID: 1789279, Thread 1789545: 
2025-07-17T11:07:25.7892691Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7892969Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7893173Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7893376Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7893877Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7894089Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7894354Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7894359Z 
2025-07-17T11:07:25.7894513Z SIGABRT(6), PID: 1789279, Thread 1789547: 
2025-07-17T11:07:25.7894986Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7895197Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7895403Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7895667Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7896226Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7896439Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7896646Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7896653Z 
2025-07-17T11:07:25.7896744Z SIGABRT(6), PID: 1789279, Thread 1789549: 
2025-07-17T11:07:25.7897215Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7897422Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7897626Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7897828Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7898322Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7898533Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7898743Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7898747Z 
2025-07-17T11:07:25.7898838Z SIGABRT(6), PID: 1789279, Thread 1789551: 
2025-07-17T11:07:25.7899307Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7899513Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7899717Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7899915Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7900416Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7900623Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7900835Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7900920Z 
2025-07-17T11:07:25.7901013Z SIGABRT(6), PID: 1789279, Thread 1789553: 
2025-07-17T11:07:25.7901483Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7901690Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7901895Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7902092Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7902654Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7902936Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7903157Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7903161Z 
2025-07-17T11:07:25.7903251Z SIGABRT(6), PID: 1789279, Thread 1789554: 
2025-07-17T11:07:25.7903810Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7904134Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7904349Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7904549Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7905054Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7905265Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7905479Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7905483Z 
2025-07-17T11:07:25.7905575Z SIGABRT(6), PID: 1789279, Thread 1789555: 
2025-07-17T11:07:25.7906053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7906261Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7906466Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7906665Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7907169Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7907377Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7907585Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7907591Z 
2025-07-17T11:07:25.7907681Z SIGABRT(6), PID: 1789279, Thread 1789557: 
2025-07-17T11:07:25.7908153Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7908364Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7908568Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7908764Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7909262Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7909541Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7909750Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7909756Z 
2025-07-17T11:07:25.7909851Z SIGABRT(6), PID: 1789279, Thread 1789558: 
2025-07-17T11:07:25.7910317Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7910525Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7910787Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7911048Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7911544Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7911754Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7912025Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7912030Z 
2025-07-17T11:07:25.7912122Z SIGABRT(6), PID: 1789279, Thread 1789559: 
2025-07-17T11:07:25.7912586Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7912796Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7913005Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7913204Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7913703Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7913915Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7914121Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7914125Z 
2025-07-17T11:07:25.7914216Z SIGABRT(6), PID: 1789279, Thread 1789560: 
2025-07-17T11:07:25.7914688Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7914895Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7915098Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7915299Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7915791Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7916002Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7916209Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7916213Z 
2025-07-17T11:07:25.7916305Z SIGABRT(6), PID: 1789279, Thread 1789561: 
2025-07-17T11:07:25.7916773Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7916982Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7917256Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7917455Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7917949Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7918160Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7918368Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7918372Z 
2025-07-17T11:07:25.7918464Z SIGABRT(6), PID: 1789279, Thread 1789562: 
2025-07-17T11:07:25.7919069Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7919281Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7919489Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7919685Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7920245Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7920455Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7920665Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7920669Z 
2025-07-17T11:07:25.7920760Z SIGABRT(6), PID: 1789279, Thread 1789563: 
2025-07-17T11:07:25.7921235Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7921439Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7921644Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7921842Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7922338Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7922546Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7922756Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7922761Z 
2025-07-17T11:07:25.7922851Z SIGABRT(6), PID: 1789279, Thread 1789564: 
2025-07-17T11:07:25.7923322Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7923527Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7923733Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7923931Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7924424Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7924629Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7924840Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7924845Z 
2025-07-17T11:07:25.7924936Z SIGABRT(6), PID: 1789279, Thread 1789565: 
2025-07-17T11:07:25.7925408Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7925682Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7925889Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7926086Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7926589Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7926855Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7927124Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7927128Z 
2025-07-17T11:07:25.7927222Z SIGABRT(6), PID: 1789279, Thread 1789566: 
2025-07-17T11:07:25.7927695Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7927898Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7928166Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7928364Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7928863Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7929068Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7929277Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7929283Z 
2025-07-17T11:07:25.7929371Z SIGABRT(6), PID: 1789279, Thread 1789567: 
2025-07-17T11:07:25.7929842Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7930049Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7930252Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7930449Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7930942Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7931152Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7931359Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7931363Z 
2025-07-17T11:07:25.7931455Z SIGABRT(6), PID: 1789279, Thread 1789568: 
2025-07-17T11:07:25.7931923Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7932130Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7932333Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7932528Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7933022Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7933234Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7933513Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7933518Z 
2025-07-17T11:07:25.7933610Z SIGABRT(6), PID: 1789279, Thread 1789569: 
2025-07-17T11:07:25.7934080Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7934287Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7934489Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7934748Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7935297Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7935512Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7935720Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7935783Z 
2025-07-17T11:07:25.7935877Z SIGABRT(6), PID: 1789279, Thread 1789570: 
2025-07-17T11:07:25.7936407Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7936614Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7936819Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7937018Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7937510Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7937721Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7937927Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7937934Z 
2025-07-17T11:07:25.7938026Z SIGABRT(6), PID: 1789279, Thread 1789571: 
2025-07-17T11:07:25.7938491Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7938697Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7938902Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7939103Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7939594Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7939805Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7940013Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7940019Z 
2025-07-17T11:07:25.7940107Z SIGABRT(6), PID: 1789279, Thread 1789572: 
2025-07-17T11:07:25.7940583Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7940787Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7940994Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7941190Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7941789Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7941998Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7942207Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7942212Z 
2025-07-17T11:07:25.7942300Z SIGABRT(6), PID: 1789279, Thread 1789573: 
2025-07-17T11:07:25.7942853Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7943064Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7943340Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7943541Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7944042Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7944319Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7944528Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7944532Z 
2025-07-17T11:07:25.7944620Z SIGABRT(6), PID: 1789279, Thread 1789574: 
2025-07-17T11:07:25.7945096Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7945303Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7945508Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7945708Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7946201Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7946410Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7946618Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7946622Z 
2025-07-17T11:07:25.7946709Z SIGABRT(6), PID: 1789279, Thread 1789575: 
2025-07-17T11:07:25.7947178Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7947385Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7947592Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7947789Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7948285Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7948489Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7948698Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7948703Z 
2025-07-17T11:07:25.7948793Z SIGABRT(6), PID: 1789279, Thread 1789576: 
2025-07-17T11:07:25.7949264Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7949561Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7949767Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7949960Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7950457Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7950662Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7950932Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7950937Z 
2025-07-17T11:07:25.7951034Z SIGABRT(6), PID: 1789279, Thread 1789577: 
2025-07-17T11:07:25.7951564Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7951778Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7951982Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7952243Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7952739Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7952953Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7953164Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7953168Z 
2025-07-17T11:07:25.7953265Z SIGABRT(6), PID: 1789279, Thread 1789578: 
2025-07-17T11:07:25.7953736Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7953945Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7954148Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7954346Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7954841Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7955052Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7955262Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7955267Z 
2025-07-17T11:07:25.7955357Z SIGABRT(6), PID: 1789279, Thread 1789579: 
2025-07-17T11:07:25.7955829Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7956037Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7956240Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7956438Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7956929Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7957140Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7957350Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7957423Z 
2025-07-17T11:07:25.7957520Z SIGABRT(6), PID: 1789279, Thread 1789580: 
2025-07-17T11:07:25.7957988Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7958201Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7958404Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7958601Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7959153Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7959424Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7959634Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7959640Z 
2025-07-17T11:07:25.7959732Z SIGABRT(6), PID: 1789279, Thread 1789581: 
2025-07-17T11:07:25.7960198Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7960468Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7960671Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7960871Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7961368Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7961578Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7961786Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7961793Z 
2025-07-17T11:07:25.7961884Z SIGABRT(6), PID: 1789279, Thread 1789582: 
2025-07-17T11:07:25.7962353Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7962558Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7962763Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7962964Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7963462Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7963670Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7963879Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7963884Z 
2025-07-17T11:07:25.7963973Z SIGABRT(6), PID: 1789279, Thread 1789583: 
2025-07-17T11:07:25.7964442Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7964645Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7964854Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7965051Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7965546Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7965828Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7966036Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7966042Z 
2025-07-17T11:07:25.7966131Z SIGABRT(6), PID: 1789279, Thread 1789584: 
2025-07-17T11:07:25.7966601Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7966807Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7967070Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7967326Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7967826Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7968037Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7968415Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7968419Z 
2025-07-17T11:07:25.7968508Z SIGABRT(6), PID: 1789279, Thread 1789585: 
2025-07-17T11:07:25.7968981Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7969187Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7969390Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7969587Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7970086Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7970294Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7970505Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7970509Z 
2025-07-17T11:07:25.7970600Z SIGABRT(6), PID: 1789279, Thread 1789586: 
2025-07-17T11:07:25.7971068Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7971274Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7971480Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7971676Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7972172Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7972383Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7972590Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7972593Z 
2025-07-17T11:07:25.7972687Z SIGABRT(6), PID: 1789279, Thread 1789587: 
2025-07-17T11:07:25.7973155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7973364Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7973565Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7973833Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7974325Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7974536Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7974743Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7974747Z 
2025-07-17T11:07:25.7974838Z SIGABRT(6), PID: 1789279, Thread 1789588: 
2025-07-17T11:07:25.7975363Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7975628Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7975837Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7976088Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7976581Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7976905Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7977115Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7977119Z 
2025-07-17T11:07:25.7977213Z SIGABRT(6), PID: 1789279, Thread 1789590: 
2025-07-17T11:07:25.7977682Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7977889Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7978092Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7978292Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7978787Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7978995Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7979201Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7979207Z 
2025-07-17T11:07:25.7979300Z SIGABRT(6), PID: 1789279, Thread 1789592: 
2025-07-17T11:07:25.7979766Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7979974Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7980177Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7980375Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7980867Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7981076Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7981282Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7981286Z 
2025-07-17T11:07:25.7981380Z SIGABRT(6), PID: 1789279, Thread 1789594: 
2025-07-17T11:07:25.7981848Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7982149Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7982353Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7982554Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7983048Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7983329Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7983541Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7983612Z 
2025-07-17T11:07:25.7983704Z SIGABRT(6), PID: 1789279, Thread 1789596: 
2025-07-17T11:07:25.7984180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7984388Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7984652Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7984849Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7985343Z frame #4: <unknown function> + 0x3475ab (0x7fa9ed4355ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.7985551Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7985762Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7985768Z 
2025-07-17T11:07:25.7985856Z SIGABRT(6), PID: 1789279, Thread 1789804: 
2025-07-17T11:07:25.7986327Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7986782Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fa8fec9508a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7986994Z frame #2: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7987186Z frame #3: pthread_kill + 0x12c (0x7fa9f059f9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7987353Z frame #4: raise + 0x16 (0x7fa9f054b476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7987514Z frame #5: abort + 0xd3 (0x7fa9f05317f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7987757Z frame #6: <unknown function> + 0x972cc (0x7fa86c81e2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7987987Z frame #7: <unknown function> + 0x95f47 (0x7fa86c81cf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7988216Z frame #8: <unknown function> + 0x34b71 (0x7fa86c7bbb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.7988426Z frame #9: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7988643Z frame #10: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7988648Z 
2025-07-17T11:07:25.7988740Z SIGABRT(6), PID: 1789279, Thread 1793117: 
2025-07-17T11:07:25.7989215Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7989421Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7989588Z frame #2: __poll + 0x4f (0x7fa9f0621bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7990066Z frame #3: <unknown function> + 0x489f (0x7fa9f012d89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.7990294Z frame #4: <unknown function> + 0x1483bc (0x56424bc2b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7990541Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7990766Z frame #6: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7991003Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7991286Z frame #8: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7991532Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x56424bc13a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7991820Z frame #10: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7992072Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x56424bc11c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7992299Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7992596Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7992821Z frame #14: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7993056Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7993280Z frame #16: <unknown function> + 0x147734 (0x56424bc2a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7993501Z frame #17: <unknown function> + 0x21ca82 (0x56424bcffa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7993724Z frame #18: <unknown function> + 0x21c9e4 (0x56424bcff9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.7993939Z frame #19: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7994154Z frame #20: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7994160Z 
2025-07-17T11:07:25.7994252Z SIGABRT(6), PID: 1789279, Thread 1793454: 
2025-07-17T11:07:25.7994733Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7994945Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7995152Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7995372Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa9f059d2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7995822Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fa92a9dd048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7996245Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fa92a9dfbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7996484Z frame #6: <unknown function> + 0xdc253 (0x7fa8fcf01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7996694Z frame #7: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7996907Z frame #8: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7996911Z 
2025-07-17T11:07:25.7997001Z SIGABRT(6), PID: 1789279, Thread 1793455: 
2025-07-17T11:07:25.7997478Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.7997683Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7997959Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7998184Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa9f059d2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7998659Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fa92a9d7741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.7998886Z frame #5: <unknown function> + 0xdc253 (0x7fa8fcf01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.7999094Z frame #6: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7999360Z frame #7: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.7999369Z 
2025-07-17T11:07:25.7999459Z SIGABRT(6), PID: 1789279, Thread 1793456: 
2025-07-17T11:07:25.7999987Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8000200Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8000370Z frame #2: ioctl + 0x3f (0x7fa9f062394f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8000668Z frame #3: <unknown function> + 0x13eef0 (0x7fa86c8c5ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8000899Z frame #4: <unknown function> + 0x1378c4 (0x7fa86c8be8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8001128Z frame #5: <unknown function> + 0x95b2b (0x7fa86c81cb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8001359Z frame #6: <unknown function> + 0x34b71 (0x7fa86c7bbb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8001570Z frame #7: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8001780Z frame #8: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8001786Z 
2025-07-17T11:07:25.8001875Z SIGABRT(6), PID: 1789279, Thread 1793553: 
2025-07-17T11:07:25.8002346Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8002552Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8002721Z frame #2: __poll + 0x4f (0x7fa9f0621bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8002934Z frame #3: <unknown function> + 0xd4540 (0x7fa8b4a35540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8003143Z frame #4: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8003350Z frame #5: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8003356Z 
2025-07-17T11:07:25.8003446Z SIGABRT(6), PID: 1789279, Thread 1793562: 
2025-07-17T11:07:25.8003918Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8004125Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8004289Z frame #2: __poll + 0x4f (0x7fa9f0621bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8004502Z frame #3: <unknown function> + 0xd65a2 (0x7fa8b4a375a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8004709Z frame #4: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8004918Z frame #5: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8004922Z 
2025-07-17T11:07:25.8005011Z SIGABRT(6), PID: 1789279, Thread 1793566: 
2025-07-17T11:07:25.8005484Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8005759Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8005966Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8006169Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8006382Z frame #4: <unknown function> + 0xd1e7f (0x7fa8b4a32e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8006587Z frame #5: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8006794Z frame #6: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8006860Z 
2025-07-17T11:07:25.8006950Z SIGABRT(6), PID: 1789279, Thread 1793579: 
2025-07-17T11:07:25.8007505Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8007716Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8007923Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8008193Z frame #3: <unknown function> + 0x9ca53 (0x7fa9f05a5a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8008454Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x56424bbf7bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8008677Z frame #5: <unknown function> + 0x1d314f (0x56424bcb614f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8008903Z frame #6: <unknown function> + 0x1d2f6b (0x56424bcb5f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8009123Z frame #7: <unknown function> + 0x6df36 (0x56424bb50f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8009368Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8009595Z frame #9: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8009838Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8010064Z frame #11: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8010300Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8010524Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8010758Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8010985Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8011221Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8011439Z frame #17: <unknown function> + 0x147734 (0x56424bc2a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8011661Z frame #18: <unknown function> + 0x21ca82 (0x56424bcffa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8011883Z frame #19: <unknown function> + 0x21c9e4 (0x56424bcff9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8012095Z frame #20: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8012306Z frame #21: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8012311Z 
2025-07-17T11:07:25.8012401Z SIGABRT(6), PID: 1789279, Thread 1793592: 
2025-07-17T11:07:25.8012871Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8013087Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8013324Z frame #2: read + 0x4c (0x7fa9f061d81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8013511Z frame #3: _Py_read + 0x4c (0x56424bcbda6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8013736Z frame #4: <unknown function> + 0x208c3b (0x56424bcebc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8013960Z frame #5: <unknown function> + 0x14648d (0x56424bc2948d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8014205Z frame #6: PyObject_VectorcallMethod + 0x85 (0x56424bc38215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8014429Z frame #7: <unknown function> + 0x208ae7 (0x56424bcebae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8014704Z frame #8: <unknown function> + 0x208472 (0x56424bceb472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8014986Z frame #9: <unknown function> + 0x1da6ee (0x56424bcbd6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8015208Z frame #10: <unknown function> + 0x1483bc (0x56424bc2b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8015450Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8015676Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8016132Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x56424bc0f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8016355Z frame #14: <unknown function> + 0x147734 (0x56424bc2a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8016604Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x56424bc11c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8016829Z frame #16: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8017067Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8017295Z frame #18: _PyFunction_Vectorcall + 0x6c (0x56424bc1f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8017532Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x56424bc0f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8017750Z frame #20: <unknown function> + 0x147734 (0x56424bc2a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8017974Z frame #21: <unknown function> + 0x21ca82 (0x56424bcffa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8018191Z frame #22: <unknown function> + 0x21c9e4 (0x56424bcff9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8018404Z frame #23: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8018616Z frame #24: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8018621Z 
2025-07-17T11:07:25.8018714Z SIGABRT(6), PID: 1789279, Thread 1793630: 
2025-07-17T11:07:25.8019197Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8019412Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8019615Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8019822Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8020196Z frame #4: <unknown function> + 0x5a02023 (0x7fa9349cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8020768Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa9349cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8021362Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa9349c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8021847Z frame #7: <unknown function> + 0x79d655 (0x7fa945eb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8022079Z frame #8: <unknown function> + 0xdc253 (0x7fa8fcf01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8022295Z frame #9: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8022507Z frame #10: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8022515Z 
2025-07-17T11:07:25.8022605Z SIGABRT(6), PID: 1789279, Thread 1793631: 
2025-07-17T11:07:25.8023160Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8023446Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8023659Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8023861Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8024242Z frame #4: <unknown function> + 0x5a02023 (0x7fa9349cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8024889Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa9349cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8025489Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa9349c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8025874Z frame #7: <unknown function> + 0x79d655 (0x7fa945eb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8026110Z frame #8: <unknown function> + 0xdc253 (0x7fa8fcf01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8026316Z frame #9: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8026535Z frame #10: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8026539Z 
2025-07-17T11:07:25.8026629Z SIGABRT(6), PID: 1789279, Thread 1793632: 
2025-07-17T11:07:25.8027105Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8027314Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8027519Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8027720Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8028094Z frame #4: <unknown function> + 0x5a02023 (0x7fa9349cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8028653Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa9349cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8029249Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa9349c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8029633Z frame #7: <unknown function> + 0x79d655 (0x7fa945eb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8029866Z frame #8: <unknown function> + 0xdc253 (0x7fa8fcf01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8030073Z frame #9: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8030365Z frame #10: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8030369Z 
2025-07-17T11:07:25.8030459Z SIGABRT(6), PID: 1789281, Thread 1789281: 
2025-07-17T11:07:25.8030935Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8031143Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8031376Z frame #2: <unknown function> + 0x7743a (0x7f25927ce43a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8031695Z frame #3: <unknown function> + 0x76c8e (0x7f25927cdc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8032007Z frame #4: <unknown function> + 0x6ad69 (0x7f25927c1d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8032227Z frame #5: <unknown function> + 0x3f54db (0x7f26254264db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8032443Z frame #6: <unknown function> + 0x3e11d8 (0x7f26254121d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8032652Z frame #7: <unknown function> + 0xb2f86 (0x7f26250e3f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8032932Z frame #8: <unknown function> + 0x8e501 (0x7f26250bf501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8033315Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f264fec67ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.8033695Z frame #10: <unknown function> + 0xd307d9 (0x7f266e4197d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8033920Z frame #11: <unknown function> + 0x139902 (0x558f71a75902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8034175Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x558f71a6ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8034401Z frame #13: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8034646Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x558f71a6ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8034871Z frame #15: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8035114Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8035335Z frame #17: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8035572Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8035791Z frame #19: <unknown function> + 0x1474b2 (0x558f71a834b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8035999Z frame #20: PyObject_Call + 0xbc (0x558f71a83e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8036220Z frame #21: <unknown function> + 0x1ff69d (0x558f71b3b69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8036447Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x558f71a71023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8036688Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x558f71a6ccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8036917Z frame #24: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8037151Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x558f71a683bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8037375Z frame #26: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8037614Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x558f71a683bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8037840Z frame #28: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8038077Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x558f71a6ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8038418Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x558f71a704b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8038637Z frame #31: _PyObject_Call_Prepend + 0x69 (0x558f71a81ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8038859Z frame #32: <unknown function> + 0x205949 (0x558f71b41949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8039080Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x558f71a71023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8039323Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x558f71a6ccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8039679Z frame #35: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8040023Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x558f71a683bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8040251Z frame #37: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8040489Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x558f71a683bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8040711Z frame #39: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8041035Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x558f71a683bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8041256Z frame #41: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8041497Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x558f71a693f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8041720Z frame #43: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8042109Z frame #44: <unknown function> + 0x7ab3f7 (0x7f266de943f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8042330Z frame #45: <unknown function> + 0x13bc88 (0x558f71a77c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8042532Z frame #46: PyObject_Call + 0x20f (0x558f71a83f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8042771Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x558f71a6d98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8042989Z frame #48: <unknown function> + 0x147608 (0x558f71a83608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8043229Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x558f71a6ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8043456Z frame #50: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8043694Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x558f71a683bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8043917Z frame #52: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8044153Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x558f71a693f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8044378Z frame #54: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8044614Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x558f71a683bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8044839Z frame #56: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8045075Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x558f71a6ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8045298Z frame #58: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8045536Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x558f71a6ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8045757Z frame #60: <unknown function> + 0x147608 (0x558f71a83608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8045998Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x558f71a6ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8046314Z frame #62: <unknown function> + 0x147608 (0x558f71a83608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8046318Z 
2025-07-17T11:07:25.8046412Z SIGABRT(6), PID: 1789281, Thread 1789725: 
2025-07-17T11:07:25.8046890Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8047108Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8047316Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8047614Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8048197Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8048419Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8048631Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8048697Z 
2025-07-17T11:07:25.8048794Z SIGABRT(6), PID: 1789281, Thread 1789726: 
2025-07-17T11:07:25.8049262Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8049472Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8049678Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8049880Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8050375Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8050587Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8050797Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8050803Z 
2025-07-17T11:07:25.8050896Z SIGABRT(6), PID: 1789279, Thread 1793633: 
2025-07-17T11:07:25.8051365Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8051574Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8051775Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8051975Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8052351Z frame #4: <unknown function> + 0x5a02023 (0x7fa9349cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8052919Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa9349cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8053511Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa9349c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8053897Z frame #7: <unknown function> + 0x79d655 (0x7fa945eb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8054127Z frame #8: <unknown function> + 0xdc253 (0x7fa8fcf01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8054341Z frame #9: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8054630Z frame #10: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8054634Z 
2025-07-17T11:07:25.8054727Z SIGABRT(6), PID: 1789281, Thread 1789727: 
2025-07-17T11:07:25.8055196Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8055412Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8055617Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8055899Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8056526Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8056745Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8056955Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8056962Z 
2025-07-17T11:07:25.8057159Z SIGABRT(6), PID: 1789281, Thread 1789728: 
2025-07-17T11:07:25.8057641Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8057846Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8058053Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8058251Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8058750Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8058958Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8059169Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8059174Z 
2025-07-17T11:07:25.8059263Z SIGABRT(6), PID: 1789281, Thread 1789729: 
2025-07-17T11:07:25.8059734Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8059939Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8060148Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8060348Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8060839Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8061048Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8061261Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8061265Z 
2025-07-17T11:07:25.8061352Z SIGABRT(6), PID: 1789281, Thread 1789730: 
2025-07-17T11:07:25.8061818Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8062027Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8062232Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8062427Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8063011Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8063220Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8063431Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8063435Z 
2025-07-17T11:07:25.8063522Z SIGABRT(6), PID: 1789279, Thread 1793634: 
2025-07-17T11:07:25.8063995Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8064268Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8064534Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8064737Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8065111Z frame #4: <unknown function> + 0x5a02023 (0x7fa9349cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8065740Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa9349cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8066340Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa9349c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8066726Z frame #7: <unknown function> + 0x79d655 (0x7fa945eb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8066962Z frame #8: <unknown function> + 0xdc253 (0x7fa8fcf01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8067171Z frame #9: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8067393Z frame #10: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8067398Z 
2025-07-17T11:07:25.8067489Z SIGABRT(6), PID: 1789281, Thread 1789731: 
2025-07-17T11:07:25.8067962Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8068171Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8068375Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8068578Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8069073Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8069288Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8069496Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8069501Z 
2025-07-17T11:07:25.8069591Z SIGABRT(6), PID: 1789281, Thread 1789732: 
2025-07-17T11:07:25.8070056Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8070261Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8070464Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8070664Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8071271Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8071482Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8071693Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8071699Z 
2025-07-17T11:07:25.8071791Z SIGABRT(6), PID: 1789281, Thread 1789733: 
2025-07-17T11:07:25.8072253Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8072522Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8072782Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8072984Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8073482Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8073748Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8073956Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8073960Z 
2025-07-17T11:07:25.8074050Z SIGABRT(6), PID: 1789281, Thread 1789734: 
2025-07-17T11:07:25.8074511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8074718Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8074923Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8075124Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8075614Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8075825Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8076031Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8076036Z 
2025-07-17T11:07:25.8076128Z SIGABRT(6), PID: 1789279, Thread 1793635: 
2025-07-17T11:07:25.8076595Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8076807Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8077011Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8077212Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8077581Z frame #4: <unknown function> + 0x5a02023 (0x7fa9349cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8078149Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa9349cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8078747Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa9349c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8079133Z frame #7: <unknown function> + 0x79d655 (0x7fa945eb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8079437Z frame #8: <unknown function> + 0xdc253 (0x7fa8fcf01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8079648Z frame #9: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8079864Z frame #10: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8079953Z SIGABRT(6), PID: 1789281, Thread 1789735: 
2025-07-17T11:07:25.8080419Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8080626Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8080896Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8081152Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8081657Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8081868Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8082145Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8082150Z 
2025-07-17T11:07:25.8082153Z 
2025-07-17T11:07:25.8082241Z SIGABRT(6), PID: 1789281, Thread 1789736: 
2025-07-17T11:07:25.8082713Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8082924Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8083133Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8083329Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8083826Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8084035Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8084244Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8084248Z 
2025-07-17T11:07:25.8084338Z SIGABRT(6), PID: 1789281, Thread 1789737: 
2025-07-17T11:07:25.8084811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8085016Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8085223Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8085420Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8085914Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8086124Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8086334Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8086338Z 
2025-07-17T11:07:25.8086426Z SIGABRT(6), PID: 1789281, Thread 1789738: 
2025-07-17T11:07:25.8086892Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8087101Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8087377Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8087574Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8088073Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8088285Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8088494Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8088498Z 
2025-07-17T11:07:25.8088651Z SIGABRT(6), PID: 1789281, Thread 1789739: 
2025-07-17T11:07:25.8089178Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8089390Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8089596Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8089796Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8090348Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8090559Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8090765Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8090771Z 
2025-07-17T11:07:25.8090862Z SIGABRT(6), PID: 1789281, Thread 1789740: 
2025-07-17T11:07:25.8091329Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8091538Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8091740Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8091943Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8092433Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8092646Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8092853Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8092857Z 
2025-07-17T11:07:25.8092949Z SIGABRT(6), PID: 1789279, Thread 1793636: 
2025-07-17T11:07:25.8093419Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8093632Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8093835Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8094034Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8094407Z frame #4: <unknown function> + 0x5a02023 (0x7fa9349cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8094976Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa9349cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8095571Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa9349c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8096096Z frame #7: <unknown function> + 0x79d655 (0x7fa945eb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8096333Z frame #8: <unknown function> + 0xdc253 (0x7fa8fcf01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8096544Z frame #9: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8096756Z frame #10: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8096761Z 
2025-07-17T11:07:25.8096852Z SIGABRT(6), PID: 1789281, Thread 1789741: 
2025-07-17T11:07:25.8097394Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8097809Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8098023Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8098225Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8098823Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8099035Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8099253Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8099257Z 
2025-07-17T11:07:25.8099349Z SIGABRT(6), PID: 1789281, Thread 1789742: 
2025-07-17T11:07:25.8099821Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8100026Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8100232Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8100427Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8100926Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8101129Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8101338Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8101342Z 
2025-07-17T11:07:25.8101429Z SIGABRT(6), PID: 1789281, Thread 1789743: 
2025-07-17T11:07:25.8101897Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8102105Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8102312Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8102513Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8103008Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8103215Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8103428Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8103432Z 
2025-07-17T11:07:25.8103523Z SIGABRT(6), PID: 1789281, Thread 1789744: 
2025-07-17T11:07:25.8103991Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8104280Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8104487Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8104684Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8105183Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8105454Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8105671Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8105735Z 
2025-07-17T11:07:25.8105827Z SIGABRT(6), PID: 1789281, Thread 1789745: 
2025-07-17T11:07:25.8106304Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8106509Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8106779Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8106976Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8107467Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8107674Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8107884Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8107890Z 
2025-07-17T11:07:25.8107979Z SIGABRT(6), PID: 1789281, Thread 1789746: 
2025-07-17T11:07:25.8108465Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8108674Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8108877Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8109082Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8109575Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8109784Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8109991Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8109999Z 
2025-07-17T11:07:25.8110093Z SIGABRT(6), PID: 1789281, Thread 1789747: 
2025-07-17T11:07:25.8110560Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8110770Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8110974Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8111172Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8111663Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8111877Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8112157Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8112161Z 
2025-07-17T11:07:25.8112255Z SIGABRT(6), PID: 1789279, Thread 1793637: 
2025-07-17T11:07:25.8112720Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa8fec94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8112929Z frame #1: <unknown function> + 0x42520 (0x7fa9f054b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8113133Z frame #2: <unknown function> + 0x91117 (0x7fa9f059a117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8113398Z frame #3: pthread_cond_wait + 0x211 (0x7fa9f059ca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8113839Z frame #4: <unknown function> + 0x5a02023 (0x7fa9349cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8114416Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa9349cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8115011Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa9349c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8115464Z frame #7: <unknown function> + 0x79d655 (0x7fa945eb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8115694Z frame #8: <unknown function> + 0xdc253 (0x7fa8fcf01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8115909Z frame #9: <unknown function> + 0x94ac3 (0x7fa9f059dac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8116124Z frame #10: <unknown function> + 0x126850 (0x7fa9f062f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8116128Z 
2025-07-17T11:07:25.8116221Z SIGABRT(6), PID: 1789281, Thread 1789748: 
2025-07-17T11:07:25.8116687Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8116898Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8117103Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8117303Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8117803Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8118013Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8118220Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8118229Z 
2025-07-17T11:07:25.8118320Z SIGABRT(6), PID: 1789281, Thread 1789749: 
2025-07-17T11:07:25.8118786Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8118991Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8119195Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8119390Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8119888Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8120097Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8120312Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8120383Z 
2025-07-17T11:07:25.8120475Z SIGABRT(6), PID: 1789281, Thread 1789750: 
2025-07-17T11:07:25.8120945Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8121150Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8121355Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8121551Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8122162Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8122379Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8122592Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8122596Z 
2025-07-17T11:07:25.8122686Z SIGABRT(6), PID: 1789281, Thread 1789751: 
2025-07-17T11:07:25.8123214Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8123419Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8123626Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8123826Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8124321Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8124533Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8124746Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8124750Z 
2025-07-17T11:07:25.8124839Z SIGABRT(6), PID: 1789281, Thread 1789752: 
2025-07-17T11:07:25.8125302Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8125510Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8125717Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8125913Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8126414Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8126622Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8126834Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8126840Z 
2025-07-17T11:07:25.8126928Z SIGABRT(6), PID: 1789281, Thread 1789753: 
2025-07-17T11:07:25.8127395Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8127600Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8127805Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8128005Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8128502Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8128782Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8128997Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8129001Z 
2025-07-17T11:07:25.8129090Z SIGABRT(6), PID: 1789281, Thread 1789754: 
2025-07-17T11:07:25.8129555Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8129821Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8130024Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8130305Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8130804Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8131018Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8131291Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8131295Z 
2025-07-17T11:07:25.8131389Z SIGABRT(6), PID: 1789281, Thread 1789755: 
2025-07-17T11:07:25.8131850Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8132059Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8132262Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8132462Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8132954Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8133165Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8133370Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8133374Z 
2025-07-17T11:07:25.8133464Z SIGABRT(6), PID: 1789281, Thread 1789756: 
2025-07-17T11:07:25.8133929Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8134138Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8134339Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8134540Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8135034Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8135248Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8135456Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8135460Z 
2025-07-17T11:07:25.8135550Z SIGABRT(6), PID: 1789281, Thread 1789757: 
2025-07-17T11:07:25.8136075Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8136288Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8136579Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8136778Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8137271Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8137482Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8137689Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8137695Z 
2025-07-17T11:07:25.8137861Z SIGABRT(6), PID: 1789281, Thread 1789758: 
2025-07-17T11:07:25.8138394Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8138607Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8138810Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8139009Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8139571Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8139782Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8139992Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8139999Z 
2025-07-17T11:07:25.8140091Z SIGABRT(6), PID: 1789281, Thread 1789759: 
2025-07-17T11:07:25.8140566Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8140773Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8140980Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8141178Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8141671Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8141877Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8142088Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8142092Z 
2025-07-17T11:07:25.8142181Z SIGABRT(6), PID: 1789281, Thread 1789760: 
2025-07-17T11:07:25.8142648Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8142854Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8143060Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8143252Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8143742Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8143952Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8144165Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8144171Z 
2025-07-17T11:07:25.8144258Z SIGABRT(6), PID: 1789281, Thread 1789761: 
2025-07-17T11:07:25.8144789Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8144994Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8145201Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8145400Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8145892Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8146158Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8146440Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8146448Z 
2025-07-17T11:07:25.8146538Z SIGABRT(6), PID: 1789281, Thread 1789762: 
2025-07-17T11:07:25.8147010Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8147273Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8147479Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8147675Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8148174Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8148383Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8148593Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8148600Z 
2025-07-17T11:07:25.8148688Z SIGABRT(6), PID: 1789281, Thread 1789763: 
2025-07-17T11:07:25.8149154Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8149360Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8149565Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8149760Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8150253Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8150459Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8150670Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8150674Z 
2025-07-17T11:07:25.8150765Z SIGABRT(6), PID: 1789281, Thread 1789764: 
2025-07-17T11:07:25.8151228Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8151435Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8151636Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8151837Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8152331Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8152615Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8152821Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8152825Z 
2025-07-17T11:07:25.8152919Z SIGABRT(6), PID: 1789281, Thread 1789765: 
2025-07-17T11:07:25.8153380Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8153587Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8153788Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8154050Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8154604Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8154817Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8155024Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8155086Z 
2025-07-17T11:07:25.8155182Z SIGABRT(6), PID: 1789281, Thread 1789766: 
2025-07-17T11:07:25.8155647Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8155854Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8156058Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8156258Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8156751Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8156967Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8157178Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8157182Z 
2025-07-17T11:07:25.8157275Z SIGABRT(6), PID: 1789281, Thread 1789767: 
2025-07-17T11:07:25.8157739Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8157947Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8158149Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8158348Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8158840Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8159050Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8159263Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8159266Z 
2025-07-17T11:07:25.8159357Z SIGABRT(6), PID: 1789281, Thread 1789768: 
2025-07-17T11:07:25.8159822Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8160032Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8160237Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8160436Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8161000Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8161212Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8161419Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8161426Z 
2025-07-17T11:07:25.8161514Z SIGABRT(6), PID: 1789281, Thread 1789769: 
2025-07-17T11:07:25.8162057Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8162321Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8162533Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8162730Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8163225Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8163491Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8163698Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8163702Z 
2025-07-17T11:07:25.8163789Z SIGABRT(6), PID: 1789281, Thread 1789770: 
2025-07-17T11:07:25.8164261Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8164470Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8164680Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8164877Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8165374Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8165581Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8165792Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8165796Z 
2025-07-17T11:07:25.8165885Z SIGABRT(6), PID: 1789281, Thread 1789771: 
2025-07-17T11:07:25.8166354Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8166558Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8166766Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8166960Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8167454Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8167659Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8167866Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8167872Z 
2025-07-17T11:07:25.8167960Z SIGABRT(6), PID: 1789281, Thread 1789772: 
2025-07-17T11:07:25.8168430Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8168712Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8168917Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8169115Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8169605Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8169810Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8170084Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8170088Z 
2025-07-17T11:07:25.8170237Z SIGABRT(6), PID: 1789281, Thread 1789773: 
2025-07-17T11:07:25.8170706Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8170917Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8171122Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8171393Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8171888Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8172103Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8172309Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8172316Z 
2025-07-17T11:07:25.8172406Z SIGABRT(6), PID: 1789281, Thread 1789774: 
2025-07-17T11:07:25.8172868Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8173076Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8173280Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8173478Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8173970Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8174180Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8174389Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8174393Z 
2025-07-17T11:07:25.8174486Z SIGABRT(6), PID: 1789281, Thread 1789775: 
2025-07-17T11:07:25.8174946Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8175156Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8175356Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8175553Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8176099Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8176318Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8176525Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8176616Z 
2025-07-17T11:07:25.8176710Z SIGABRT(6), PID: 1789281, Thread 1789776: 
2025-07-17T11:07:25.8177176Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8177387Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8177590Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8177794Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8178395Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8178684Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8178897Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8178901Z 
2025-07-17T11:07:25.8178995Z SIGABRT(6), PID: 1789281, Thread 1789777: 
2025-07-17T11:07:25.8179461Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8179751Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8179953Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8180154Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8180648Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8180858Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8181067Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8181071Z 
2025-07-17T11:07:25.8181163Z SIGABRT(6), PID: 1789281, Thread 1789778: 
2025-07-17T11:07:25.8181627Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8181833Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8182038Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8182236Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8182738Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8182946Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8183155Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8183160Z 
2025-07-17T11:07:25.8183255Z SIGABRT(6), PID: 1789281, Thread 1789779: 
2025-07-17T11:07:25.8183725Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8183932Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8184141Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8184335Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8184834Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8185110Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8185320Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8185326Z 
2025-07-17T11:07:25.8185412Z SIGABRT(6), PID: 1789281, Thread 1789780: 
2025-07-17T11:07:25.8185880Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8186085Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8186350Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8186604Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8187101Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8187311Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8187586Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8187590Z 
2025-07-17T11:07:25.8187683Z SIGABRT(6), PID: 1789281, Thread 1789781: 
2025-07-17T11:07:25.8188151Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8188355Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8188561Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8188755Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8189252Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8189461Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8189670Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8189674Z 
2025-07-17T11:07:25.8189765Z SIGABRT(6), PID: 1789281, Thread 1789782: 
2025-07-17T11:07:25.8190237Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8190440Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8190645Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8190841Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8191338Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8191546Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8191760Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8191764Z 
2025-07-17T11:07:25.8191851Z SIGABRT(6), PID: 1789281, Thread 1789783: 
2025-07-17T11:07:25.8192323Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8192532Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8192825Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8193026Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8193523Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8193732Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8193939Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8193943Z 
2025-07-17T11:07:25.8194037Z SIGABRT(6), PID: 1789281, Thread 1789784: 
2025-07-17T11:07:25.8194612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8194825Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8195032Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8195229Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8195781Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8195988Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8196192Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8196196Z 
2025-07-17T11:07:25.8196290Z SIGABRT(6), PID: 1789281, Thread 1789785: 
2025-07-17T11:07:25.8196756Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8196966Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8197169Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8197370Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8197864Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8198075Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8198282Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8198286Z 
2025-07-17T11:07:25.8198378Z SIGABRT(6), PID: 1789281, Thread 1789786: 
2025-07-17T11:07:25.8198842Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8199056Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8199258Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8199461Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8199951Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8200160Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8200374Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8200378Z 
2025-07-17T11:07:25.8200474Z SIGABRT(6), PID: 1789281, Thread 1789787: 
2025-07-17T11:07:25.8200937Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8201214Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8201418Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8201622Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8202113Z frame #4: <unknown function> + 0x3475ab (0x7f26eb4075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8202379Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8202643Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8202652Z 
2025-07-17T11:07:25.8202744Z SIGABRT(6), PID: 1789281, Thread 1789796: 
2025-07-17T11:07:25.8203222Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8203673Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f2626c6508a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8203947Z frame #2: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8204145Z frame #3: pthread_kill + 0x12c (0x7f26ee5719fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8204313Z frame #4: raise + 0x16 (0x7f26ee51d476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8204476Z frame #5: abort + 0xd3 (0x7f26ee5037f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8204721Z frame #6: <unknown function> + 0x972cc (0x7f25927ee2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8204951Z frame #7: <unknown function> + 0x95f47 (0x7f25927ecf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8205180Z frame #8: <unknown function> + 0x34b71 (0x7f259278bb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8205387Z frame #9: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8205607Z frame #10: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8205611Z 
2025-07-17T11:07:25.8205701Z SIGABRT(6), PID: 1789281, Thread 1792520: 
2025-07-17T11:07:25.8206180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8206389Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8206563Z frame #2: __poll + 0x4f (0x7f26ee5f3bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8206971Z frame #3: <unknown function> + 0x489f (0x7f26ee0ff89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.8207205Z frame #4: <unknown function> + 0x1483bc (0x558f71a843bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8207456Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8207686Z frame #6: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8207926Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8208155Z frame #8: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8208402Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x558f71a6ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8208641Z frame #10: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8208959Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x558f71a6ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8209189Z frame #12: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8209427Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8209656Z frame #14: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8209891Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8210110Z frame #16: <unknown function> + 0x147734 (0x558f71a83734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8210385Z frame #17: <unknown function> + 0x21ca82 (0x558f71b58a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8210671Z frame #18: <unknown function> + 0x21c9e4 (0x558f71b589e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8210899Z frame #19: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8211115Z frame #20: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8211119Z 
2025-07-17T11:07:25.8211270Z SIGABRT(6), PID: 1789281, Thread 1793532: 
2025-07-17T11:07:25.8211748Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8211961Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8212173Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8212395Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f26ee56f2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8212849Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f26529ad048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.8213274Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f26529afbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.8213505Z frame #6: <unknown function> + 0xdc253 (0x7f2624ed1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8213714Z frame #7: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8213926Z frame #8: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8213930Z 
2025-07-17T11:07:25.8214019Z SIGABRT(6), PID: 1789281, Thread 1793533: 
2025-07-17T11:07:25.8214494Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8214704Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8214913Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8215134Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f26ee56f2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8215608Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f26529a7741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.8215837Z frame #5: <unknown function> + 0xdc253 (0x7f2624ed1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8216097Z frame #6: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8216308Z frame #7: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8216312Z 
2025-07-17T11:07:25.8216408Z SIGABRT(6), PID: 1789281, Thread 1793534: 
2025-07-17T11:07:25.8216875Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8217172Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8217339Z frame #2: ioctl + 0x3f (0x7f26ee5f594f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8217577Z frame #3: <unknown function> + 0x13eef0 (0x7f2592895ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8217806Z frame #4: <unknown function> + 0x1378c4 (0x7f259288e8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8218038Z frame #5: <unknown function> + 0x95b2b (0x7f25927ecb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8218331Z frame #6: <unknown function> + 0x34b71 (0x7f259278bb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8218547Z frame #7: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8218825Z frame #8: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8218832Z 
2025-07-17T11:07:25.8218928Z SIGABRT(6), PID: 1789281, Thread 1793551: 
2025-07-17T11:07:25.8219396Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8219699Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8219864Z frame #2: __poll + 0x4f (0x7f26ee5f3bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8220081Z frame #3: <unknown function> + 0xd4540 (0x7f25daa05540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8220287Z frame #4: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8220501Z frame #5: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8220505Z 
2025-07-17T11:07:25.8220596Z SIGABRT(6), PID: 1789281, Thread 1793559: 
2025-07-17T11:07:25.8221067Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8221278Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8221447Z frame #2: __poll + 0x4f (0x7f26ee5f3bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8221658Z frame #3: <unknown function> + 0xd65a2 (0x7f25daa075a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8221866Z frame #4: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8222071Z frame #5: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8222079Z 
2025-07-17T11:07:25.8222168Z SIGABRT(6), PID: 1789281, Thread 1793572: 
2025-07-17T11:07:25.8222639Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8222846Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8223053Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8223254Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8223469Z frame #4: <unknown function> + 0xd1e7f (0x7f25daa02e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8223674Z frame #5: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8223883Z frame #6: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8223887Z 
2025-07-17T11:07:25.8223978Z SIGABRT(6), PID: 1789281, Thread 1793577: 
2025-07-17T11:07:25.8229344Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8229749Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8229981Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8230203Z frame #3: <unknown function> + 0x9ca53 (0x7f26ee577a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8230474Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x558f71a50bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8230714Z frame #5: <unknown function> + 0x1d314f (0x558f71b0f14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8230942Z frame #6: <unknown function> + 0x1d2f6b (0x558f71b0ef6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8231341Z frame #7: <unknown function> + 0x6df36 (0x558f719a9f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8231659Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8231891Z frame #9: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8232140Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8232372Z frame #11: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8232682Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8232904Z frame #13: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8233139Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8233364Z frame #15: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8233599Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8233818Z frame #17: <unknown function> + 0x147734 (0x558f71a83734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8234041Z frame #18: <unknown function> + 0x21ca82 (0x558f71b58a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8234260Z frame #19: <unknown function> + 0x21c9e4 (0x558f71b589e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8234487Z frame #20: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8234700Z frame #21: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8234707Z 
2025-07-17T11:07:25.8234808Z SIGABRT(6), PID: 1789281, Thread 1793589: 
2025-07-17T11:07:25.8235313Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8235537Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8235707Z frame #2: read + 0x4c (0x7f26ee5ef81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8235895Z frame #3: _Py_read + 0x4c (0x558f71b16a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8236122Z frame #4: <unknown function> + 0x208c3b (0x558f71b44c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8236345Z frame #5: <unknown function> + 0x14648d (0x558f71a8248d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8236594Z frame #6: PyObject_VectorcallMethod + 0x85 (0x558f71a91215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8236817Z frame #7: <unknown function> + 0x208ae7 (0x558f71b44ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8237031Z frame #8: <unknown function> + 0x208472 (0x558f71b44472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8237253Z frame #9: <unknown function> + 0x1da6ee (0x558f71b166ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8237475Z frame #10: <unknown function> + 0x1483bc (0x558f71a843bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8237792Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8238015Z frame #12: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8238254Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x558f71a683bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8238471Z frame #14: <unknown function> + 0x147734 (0x558f71a83734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8238723Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x558f71a6ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8239011Z frame #16: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8239253Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8239532Z frame #18: _PyFunction_Vectorcall + 0x6c (0x558f71a780ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8239772Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x558f71a687b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8239988Z frame #20: <unknown function> + 0x147734 (0x558f71a83734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8240271Z frame #21: <unknown function> + 0x21ca82 (0x558f71b58a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8240488Z frame #22: <unknown function> + 0x21c9e4 (0x558f71b589e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8240705Z frame #23: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8240915Z frame #24: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8240921Z 
2025-07-17T11:07:25.8241019Z SIGABRT(6), PID: 1789281, Thread 1793622: 
2025-07-17T11:07:25.8241500Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8241723Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8241930Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8242137Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8242518Z frame #4: <unknown function> + 0x5a02023 (0x7f265c99b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8243094Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f265c99f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8243690Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f265c9961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8244092Z frame #7: <unknown function> + 0x79d655 (0x7f266de86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8244329Z frame #8: <unknown function> + 0xdc253 (0x7f2624ed1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8244549Z frame #9: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8244764Z frame #10: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8244768Z 
2025-07-17T11:07:25.8244865Z SIGABRT(6), PID: 1789281, Thread 1793623: 
2025-07-17T11:07:25.8245349Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8245569Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8245772Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8246050Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8246428Z frame #4: <unknown function> + 0x5a02023 (0x7f265c99b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8246997Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f265c99f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8247650Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f265c9961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8248095Z frame #7: <unknown function> + 0x79d655 (0x7f266de86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8248334Z frame #8: <unknown function> + 0xdc253 (0x7f2624ed1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8248545Z frame #9: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8248758Z frame #10: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8248826Z 
2025-07-17T11:07:25.8248921Z SIGABRT(6), PID: 1789281, Thread 1793624: 
2025-07-17T11:07:25.8249398Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8249607Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8249812Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8250017Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8250390Z frame #4: <unknown function> + 0x5a02023 (0x7f265c99b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8250949Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f265c99f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8251538Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f265c9961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8251911Z frame #7: <unknown function> + 0x79d655 (0x7f266de86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8252144Z frame #8: <unknown function> + 0xdc253 (0x7f2624ed1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8252355Z frame #9: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8252571Z frame #10: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8252575Z 
2025-07-17T11:07:25.8252667Z SIGABRT(6), PID: 1789281, Thread 1793625: 
2025-07-17T11:07:25.8253141Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8253349Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8253555Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8253755Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8254123Z frame #4: <unknown function> + 0x5a02023 (0x7f265c99b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8254677Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f265c99f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8255342Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f265c9961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8255717Z frame #7: <unknown function> + 0x79d655 (0x7f266de86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8256026Z frame #8: <unknown function> + 0xdc253 (0x7f2624ed1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8256315Z frame #9: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8256631Z frame #10: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8256637Z 
2025-07-17T11:07:25.8256741Z SIGABRT(6), PID: 1789281, Thread 1793626: 
2025-07-17T11:07:25.8257242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8257536Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8257754Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8257959Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8258344Z frame #4: <unknown function> + 0x5a02023 (0x7f265c99b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8258915Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f265c99f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8259506Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f265c9961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8259889Z frame #7: <unknown function> + 0x79d655 (0x7f266de86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8260122Z frame #8: <unknown function> + 0xdc253 (0x7f2624ed1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8260338Z frame #9: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8260554Z frame #10: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8260560Z 
2025-07-17T11:07:25.8260652Z SIGABRT(6), PID: 1789281, Thread 1793627: 
2025-07-17T11:07:25.8261134Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8261347Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8261551Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8261755Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8262124Z frame #4: <unknown function> + 0x5a02023 (0x7f265c99b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8262687Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f265c99f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8263275Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f265c9961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8263768Z frame #7: <unknown function> + 0x79d655 (0x7f266de86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8263997Z frame #8: <unknown function> + 0xdc253 (0x7f2624ed1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8264210Z frame #9: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8264420Z frame #10: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8264425Z 
2025-07-17T11:07:25.8264517Z SIGABRT(6), PID: 1789281, Thread 1793628: 
2025-07-17T11:07:25.8265044Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8265319Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8265527Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8265733Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8266099Z frame #4: <unknown function> + 0x5a02023 (0x7f265c99b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8266728Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f265c99f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8267319Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f265c9961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8267696Z frame #7: <unknown function> + 0x79d655 (0x7f266de86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8267928Z frame #8: <unknown function> + 0xdc253 (0x7f2624ed1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8268136Z frame #9: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8268349Z frame #10: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8268355Z 
2025-07-17T11:07:25.8268446Z SIGABRT(6), PID: 1789281, Thread 1793629: 
2025-07-17T11:07:25.8268927Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2626c64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8269135Z frame #1: <unknown function> + 0x42520 (0x7f26ee51d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8269343Z frame #2: <unknown function> + 0x91117 (0x7f26ee56c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8269544Z frame #3: pthread_cond_wait + 0x211 (0x7f26ee56ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8269920Z frame #4: <unknown function> + 0x5a02023 (0x7f265c99b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8270479Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f265c99f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8271068Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f265c9961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8271449Z frame #7: <unknown function> + 0x79d655 (0x7f266de86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8271678Z frame #8: <unknown function> + 0xdc253 (0x7f2624ed1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8271888Z frame #9: <unknown function> + 0x94ac3 (0x7f26ee56fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8272176Z frame #10: <unknown function> + 0x126850 (0x7f26ee601850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8272180Z 
2025-07-17T11:07:25.8272279Z ('RERUN', {'yellow': True}) [28.6001s] [ 39%]
2025-07-17T11:07:25.8272933Z distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph I0717 11:00:37.857000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1794246
2025-07-17T11:07:25.8273293Z I0717 11:00:37.858000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1794247
2025-07-17T11:07:25.8273705Z I0717 11:00:37.860000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1794248
2025-07-17T11:07:25.8274113Z I0717 11:00:37.861000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1794249
2025-07-17T11:07:25.8274461Z I0717 11:00:37.862000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1794250
2025-07-17T11:07:25.8274803Z I0717 11:00:37.863000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1794251
2025-07-17T11:07:25.8275203Z I0717 11:00:37.865000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1794252
2025-07-17T11:07:25.8275543Z I0717 11:00:37.866000 1749280 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1794253
2025-07-17T11:07:25.8276557Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.8276644Z   import pkg_resources
2025-07-17T11:07:25.8277620Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.8277703Z   import pkg_resources
2025-07-17T11:07:25.8278670Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.8278751Z   import pkg_resources
2025-07-17T11:07:25.8279714Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.8279798Z   import pkg_resources
2025-07-17T11:07:25.8280762Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.8280842Z   import pkg_resources
2025-07-17T11:07:25.8281809Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.8281890Z   import pkg_resources
2025-07-17T11:07:25.8282862Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.8283014Z   import pkg_resources
2025-07-17T11:07:25.8283976Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:25.8284054Z   import pkg_resources
2025-07-17T11:07:25.8284663Z [rank7]:W0717 11:00:59.207000 1794253 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.8285140Z [rank1]:W0717 11:00:59.211000 1794247 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.8285610Z [rank5]:W0717 11:00:59.214000 1794251 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.8286134Z [rank0]:W0717 11:00:59.214000 1794246 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.8286599Z [rank2]:W0717 11:00:59.215000 1794248 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.8287064Z [rank3]:W0717 11:00:59.215000 1794249 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.8287526Z [rank6]:W0717 11:00:59.215000 1794252 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.8287990Z [rank4]:W0717 11:00:59.215000 1794250 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:25.8288279Z Memory access fault by GPU node-4 (Agent handle: 0x55dbffd6d0e0) on address 0x7f74dbe06000. Reason: Unknown.
2025-07-17T11:07:25.8288549Z Memory access fault by GPU node-9 (Agent handle: 0x55a1bb5f3290) on address 0x7f902961d000. Reason: Unknown.
2025-07-17T11:07:25.8288815Z Memory access fault by GPU node-7 (Agent handle: 0x55d54846e440) on address 0x7f5689c25000. Reason: Unknown.
2025-07-17T11:07:25.8289089Z Memory access fault by GPU node-3 (Agent handle: 0x5630685c2650) on address 0x7f4cb1c03000. Reason: Unknown.
2025-07-17T11:07:25.8289367Z Memory access fault by GPU node-6 (Agent handle: 0x55c901e05910) on address 0x7fc2b341e000. Reason: Unknown.
2025-07-17T11:07:25.8289629Z Memory access fault by GPU node-5 (Agent handle: 0x555e05822270) on address 0x7f3bc3405000. Reason: Unknown.
2025-07-17T11:07:25.8289895Z Memory access fault by GPU node-2 (Agent handle: 0x55a9d5ec0ac0) on address 0x7fc5d341d000. Reason: Unknown.
2025-07-17T11:07:25.8290162Z Memory access fault by GPU node-8 (Agent handle: 0x55891b8378a0) on address 0x7fd7c3e1f000. Reason: Unknown.
2025-07-17T11:07:25.8290265Z SIGABRT(6), PID: 1794253, Thread 1794253: 
2025-07-17T11:07:25.8290757Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8290984Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8291224Z frame #2: <unknown function> + 0x7741f (0x7f9861a5941f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8291458Z frame #3: <unknown function> + 0x76c8e (0x7f9861a58c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8291758Z frame #4: <unknown function> + 0x6ad69 (0x7f9861a4cd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8291984Z frame #5: <unknown function> + 0x3f54db (0x7f98f26b14db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8292201Z frame #6: <unknown function> + 0x3e11d8 (0x7f98f269d1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8292416Z frame #7: <unknown function> + 0xb2f86 (0x7f98f236ef86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8292624Z frame #8: <unknown function> + 0x8e501 (0x7f98f234a501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8293096Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f991d1517ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.8293548Z frame #10: <unknown function> + 0xd307d9 (0x7f993b6a47d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8293789Z frame #11: <unknown function> + 0x139902 (0x55a1b140b902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8294041Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55a1b1402a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8294270Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8294578Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55a1b1402a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8294809Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8295052Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8295279Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8295520Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8295741Z frame #19: <unknown function> + 0x1474b2 (0x55a1b14194b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8296003Z frame #20: PyObject_Call + 0xbc (0x55a1b1419e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8296231Z frame #21: <unknown function> + 0x1ff69d (0x55a1b14d169d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8296458Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55a1b1407023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8296701Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55a1b1402cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8296926Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8297168Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55a1b13fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8297394Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8297630Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55a1b13fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8297854Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8298097Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55a1b1402a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8298354Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55a1b14064b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8298574Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55a1b1417ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8298792Z frame #32: <unknown function> + 0x205949 (0x55a1b14d7949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8299017Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55a1b1407023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8299256Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55a1b1402cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8299483Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8299810Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55a1b13fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8300036Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8300275Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55a1b13fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8300499Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8300734Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55a1b13fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8301037Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8301356Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55a1b13ff3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8301585Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8301974Z frame #44: <unknown function> + 0x7ab3f7 (0x7f993b11f3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8302199Z frame #45: <unknown function> + 0x13bc88 (0x55a1b140dc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8302477Z frame #46: PyObject_Call + 0x20f (0x55a1b1419f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8302724Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55a1b140398d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8302942Z frame #48: <unknown function> + 0x147608 (0x55a1b1419608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8303186Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a1b1400c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8303413Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8303656Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55a1b13fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8303891Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8304139Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55a1b13ff3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8304367Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8304606Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55a1b13fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8304831Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8305073Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a1b1400c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8305298Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8305540Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a1b1400c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8305760Z frame #60: <unknown function> + 0x147608 (0x55a1b1419608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8305998Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a1b1400c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8306217Z frame #62: <unknown function> + 0x147608 (0x55a1b1419608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8306223Z 
2025-07-17T11:07:25.8306323Z SIGABRT(6), PID: 1794253, Thread 1794627: 
2025-07-17T11:07:25.8306808Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8307030Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8307238Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8307445Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8308025Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8308239Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8308452Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8308456Z 
2025-07-17T11:07:25.8308553Z SIGABRT(6), PID: 1794253, Thread 1794630: 
2025-07-17T11:07:25.8309093Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8309304Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8309571Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8309776Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8310278Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8310546Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8310759Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8310763Z 
2025-07-17T11:07:25.8310855Z SIGABRT(6), PID: 1794253, Thread 1794633: 
2025-07-17T11:07:25.8311332Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8311539Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8311745Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8311939Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8312433Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8312641Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8312851Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8312855Z 
2025-07-17T11:07:25.8312948Z SIGABRT(6), PID: 1794248, Thread 1794248: 
2025-07-17T11:07:25.8313429Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8313634Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8313871Z frame #2: <unknown function> + 0x7744b (0x7f7d1f62b44b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8314102Z frame #3: <unknown function> + 0x76c8e (0x7f7d1f62ac8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8314331Z frame #4: <unknown function> + 0x6ad69 (0x7f7d1f61ed69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8314545Z frame #5: <unknown function> + 0x3f54db (0x7f7db02834db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8314758Z frame #6: <unknown function> + 0x3e11d8 (0x7f7db026f1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8314970Z frame #7: <unknown function> + 0xb2f86 (0x7f7daff40f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8315181Z frame #8: <unknown function> + 0x8e501 (0x7f7daff1c501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8315567Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f7ddad237ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.8316029Z frame #10: <unknown function> + 0xd307d9 (0x7f7df92767d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8316257Z frame #11: <unknown function> + 0x139902 (0x55dbf5126902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8316512Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55dbf511da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8316741Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8316985Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55dbf511da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8317274Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8317580Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8317811Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8318050Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8318270Z frame #19: <unknown function> + 0x1474b2 (0x55dbf51344b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8318535Z frame #20: PyObject_Call + 0xbc (0x55dbf5134e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8318759Z frame #21: <unknown function> + 0x1ff69d (0x55dbf51ec69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8318985Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55dbf5122023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8319229Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55dbf511dcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8319457Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8319694Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55dbf51193bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8319922Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8320156Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55dbf51193bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8320382Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8320619Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55dbf511da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8320876Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55dbf51214b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8321097Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55dbf5132ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8321319Z frame #32: <unknown function> + 0x205949 (0x55dbf51f2949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8321542Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55dbf5122023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8321785Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55dbf511dcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8322009Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8322248Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55dbf51193bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8322472Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8322708Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55dbf51193bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8322932Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8323172Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55dbf51193bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8323394Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8323731Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55dbf511a3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8323955Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8324349Z frame #44: <unknown function> + 0x7ab3f7 (0x7f7df8cf13f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8324569Z frame #45: <unknown function> + 0x13bc88 (0x55dbf5128c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8324776Z frame #46: PyObject_Call + 0x20f (0x55dbf5134f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8325081Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55dbf511e98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8325359Z frame #48: <unknown function> + 0x147608 (0x55dbf5134608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8325602Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55dbf511bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8325831Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8326068Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55dbf51193bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8326352Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8326590Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55dbf511a3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8326814Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8327052Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55dbf51193bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8327280Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8327520Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55dbf511bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8327746Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8327982Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55dbf511bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8328206Z frame #60: <unknown function> + 0x147608 (0x55dbf5134608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8328444Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55dbf511bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8328665Z frame #62: <unknown function> + 0x147608 (0x55dbf5134608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8328669Z 
2025-07-17T11:07:25.8328764Z SIGABRT(6), PID: 1794253, Thread 1794636: 
2025-07-17T11:07:25.8329251Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8329467Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8329677Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8329878Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8330378Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8330585Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8330798Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8330888Z SIGABRT(6), PID: 1794248, Thread 1794352: 
2025-07-17T11:07:25.8331362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8331638Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8331844Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8332046Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8332549Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8332755Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8333035Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8333040Z 
2025-07-17T11:07:25.8333044Z 
2025-07-17T11:07:25.8333196Z SIGABRT(6), PID: 1794253, Thread 1794639: 
2025-07-17T11:07:25.8333677Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8333885Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8334156Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8334353Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8334846Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8335055Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8335265Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8335272Z 
2025-07-17T11:07:25.8335362Z SIGABRT(6), PID: 1794248, Thread 1794355: 
2025-07-17T11:07:25.8335832Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8336100Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8336306Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8336513Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8337005Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8337218Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8337429Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8337437Z 
2025-07-17T11:07:25.8337528Z SIGABRT(6), PID: 1794253, Thread 1794642: 
2025-07-17T11:07:25.8337993Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8338202Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8338405Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8338602Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8339094Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8339305Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8339512Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8339691Z SIGABRT(6), PID: 1794248, Thread 1794359: 
2025-07-17T11:07:25.8340158Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8340373Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8340574Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8340771Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8341334Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8341625Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8341843Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8341847Z 
2025-07-17T11:07:25.8341850Z 
2025-07-17T11:07:25.8341944Z SIGABRT(6), PID: 1794253, Thread 1794645: 
2025-07-17T11:07:25.8342408Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8342710Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8342915Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8343116Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8343607Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8343818Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8344027Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8344031Z 
2025-07-17T11:07:25.8344129Z SIGABRT(6), PID: 1794248, Thread 1794363: 
2025-07-17T11:07:25.8344597Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8344804Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8345005Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8345209Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8345701Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8345916Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8346124Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8346133Z 
2025-07-17T11:07:25.8346221Z SIGABRT(6), PID: 1794253, Thread 1794648: 
2025-07-17T11:07:25.8346691Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8346894Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8347101Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8347298Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8347793Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8348077Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8348289Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8348294Z 
2025-07-17T11:07:25.8348382Z SIGABRT(6), PID: 1794248, Thread 1794365: 
2025-07-17T11:07:25.8348853Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8349121Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8349330Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8349586Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8350087Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8350296Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8350590Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8350594Z 
2025-07-17T11:07:25.8350684Z SIGABRT(6), PID: 1794253, Thread 1794651: 
2025-07-17T11:07:25.8351154Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8351359Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8351565Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8351761Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8352255Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8352464Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8352675Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8352679Z 
2025-07-17T11:07:25.8352768Z SIGABRT(6), PID: 1794248, Thread 1794367: 
2025-07-17T11:07:25.8353237Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8353442Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8353649Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8353849Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8354344Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8354554Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8354765Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8354769Z 
2025-07-17T11:07:25.8354858Z SIGABRT(6), PID: 1794253, Thread 1794654: 
2025-07-17T11:07:25.8355328Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8355534Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8355815Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8356011Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8356504Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8356715Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8356924Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8356929Z 
2025-07-17T11:07:25.8357081Z SIGABRT(6), PID: 1794248, Thread 1794369: 
2025-07-17T11:07:25.8357620Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8357831Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8358034Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8358232Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8358788Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8358999Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8359208Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8359214Z 
2025-07-17T11:07:25.8359305Z SIGABRT(6), PID: 1794253, Thread 1794657: 
2025-07-17T11:07:25.8359774Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8359986Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8360190Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8360389Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8360877Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8361084Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8361292Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8361296Z 
2025-07-17T11:07:25.8361387Z SIGABRT(6), PID: 1794253, Thread 1794660: 
2025-07-17T11:07:25.8361854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8362062Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8362263Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8362463Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8362951Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8363161Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8363368Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8363460Z SIGABRT(6), PID: 1794248, Thread 1794371: 
2025-07-17T11:07:25.8363929Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8364323Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8364527Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8364725Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8365219Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8365496Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8365767Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8365772Z 
2025-07-17T11:07:25.8365775Z 
2025-07-17T11:07:25.8365871Z SIGABRT(6), PID: 1794253, Thread 1794663: 
2025-07-17T11:07:25.8366341Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8366610Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8366812Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8367014Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8367523Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8367735Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8367950Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8367956Z 
2025-07-17T11:07:25.8368045Z SIGABRT(6), PID: 1794248, Thread 1794373: 
2025-07-17T11:07:25.8368523Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8368733Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8368942Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8369139Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8369644Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8369856Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8370070Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8370074Z 
2025-07-17T11:07:25.8370164Z SIGABRT(6), PID: 1794248, Thread 1794375: 
2025-07-17T11:07:25.8370631Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8370840Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8371043Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8371241Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8371741Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8371947Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8372236Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8372240Z 
2025-07-17T11:07:25.8372329Z SIGABRT(6), PID: 1794253, Thread 1794666: 
2025-07-17T11:07:25.8372805Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8373009Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8373215Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8373475Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8374032Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8374248Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8374462Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8374526Z 
2025-07-17T11:07:25.8374617Z SIGABRT(6), PID: 1794248, Thread 1794377: 
2025-07-17T11:07:25.8375089Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8375294Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8375502Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8375700Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8376251Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8376461Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8376674Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8376680Z 
2025-07-17T11:07:25.8376769Z SIGABRT(6), PID: 1794253, Thread 1794669: 
2025-07-17T11:07:25.8377241Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8377449Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8377651Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8377854Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8378348Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8378558Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8378769Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8378861Z SIGABRT(6), PID: 1794248, Thread 1794379: 
2025-07-17T11:07:25.8379327Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8379535Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8379740Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8379943Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8380548Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8380768Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8380982Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8380987Z 
2025-07-17T11:07:25.8380990Z 
2025-07-17T11:07:25.8381083Z SIGABRT(6), PID: 1794253, Thread 1794672: 
2025-07-17T11:07:25.8381632Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8381846Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8382147Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8382367Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8382868Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8383155Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8383365Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8383458Z SIGABRT(6), PID: 1794248, Thread 1794381: 
2025-07-17T11:07:25.8383934Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8384144Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8384351Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8384558Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8385052Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8385260Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8385469Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8385473Z 
2025-07-17T11:07:25.8385477Z 
2025-07-17T11:07:25.8385569Z SIGABRT(6), PID: 1794253, Thread 1794675: 
2025-07-17T11:07:25.8386044Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8386250Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8386453Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8386652Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8387149Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8387358Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8387566Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8387573Z 
2025-07-17T11:07:25.8387663Z SIGABRT(6), PID: 1794248, Thread 1794383: 
2025-07-17T11:07:25.8388138Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8388415Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8388623Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8388824Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8391072Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8391355Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8391591Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8391602Z 
2025-07-17T11:07:25.8391707Z SIGABRT(6), PID: 1794253, Thread 1794678: 
2025-07-17T11:07:25.8392307Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8392545Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8392758Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8393036Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8393582Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8393802Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8394013Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8394020Z 
2025-07-17T11:07:25.8394120Z SIGABRT(6), PID: 1794248, Thread 1794385: 
2025-07-17T11:07:25.8394608Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8394818Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8395028Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8395230Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8395735Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8395946Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8396160Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8396164Z 
2025-07-17T11:07:25.8396256Z SIGABRT(6), PID: 1794253, Thread 1794681: 
2025-07-17T11:07:25.8396737Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8396940Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8397144Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8397344Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8397847Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8398055Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8398267Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8398337Z 
2025-07-17T11:07:25.8398434Z SIGABRT(6), PID: 1794248, Thread 1794387: 
2025-07-17T11:07:25.8398903Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8399111Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8399319Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8399618Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8400123Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8400396Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8400615Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8400622Z 
2025-07-17T11:07:25.8400713Z SIGABRT(6), PID: 1794253, Thread 1794684: 
2025-07-17T11:07:25.8401189Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8401461Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8401672Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8401872Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8402370Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8402576Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8402791Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8402795Z 
2025-07-17T11:07:25.8402885Z SIGABRT(6), PID: 1794248, Thread 1794389: 
2025-07-17T11:07:25.8403359Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8403566Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8403778Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8403973Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8404465Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8404674Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8404885Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8404891Z 
2025-07-17T11:07:25.8404980Z SIGABRT(6), PID: 1794253, Thread 1794686: 
2025-07-17T11:07:25.8405451Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8405661Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8405863Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8406060Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8406553Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8406829Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8407040Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8407046Z 
2025-07-17T11:07:25.8407138Z SIGABRT(6), PID: 1794248, Thread 1794391: 
2025-07-17T11:07:25.8407673Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8407889Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8408093Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8408932Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8409438Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8409653Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8409931Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8409938Z 
2025-07-17T11:07:25.8410033Z SIGABRT(6), PID: 1794253, Thread 1794689: 
2025-07-17T11:07:25.8410500Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8410712Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8410915Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8411114Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8411610Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8411817Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8412025Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8412029Z 
2025-07-17T11:07:25.8412123Z SIGABRT(6), PID: 1794248, Thread 1794393: 
2025-07-17T11:07:25.8412593Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8412802Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8413005Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8413208Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8413707Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8413923Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8414133Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8414228Z SIGABRT(6), PID: 1794253, Thread 1794692: 
2025-07-17T11:07:25.8414694Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8414903Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8415105Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8415377Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8415871Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8416158Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8416466Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8416471Z 
2025-07-17T11:07:25.8416479Z 
2025-07-17T11:07:25.8416576Z SIGABRT(6), PID: 1794248, Thread 1794395: 
2025-07-17T11:07:25.8417055Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8417343Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8417558Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8417757Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8418333Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8418541Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8418750Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8418754Z 
2025-07-17T11:07:25.8418842Z SIGABRT(6), PID: 1794253, Thread 1794695: 
2025-07-17T11:07:25.8419319Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8419527Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8419731Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8419927Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8420421Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8420629Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8420839Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8420844Z 
2025-07-17T11:07:25.8420933Z SIGABRT(6), PID: 1794248, Thread 1794397: 
2025-07-17T11:07:25.8421401Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8421608Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8421815Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8422016Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8422515Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8422723Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8422936Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8422939Z 
2025-07-17T11:07:25.8423030Z SIGABRT(6), PID: 1794253, Thread 1794698: 
2025-07-17T11:07:25.8423503Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8423793Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8424001Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8424198Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8424762Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8424972Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8425272Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8425277Z 
2025-07-17T11:07:25.8425374Z SIGABRT(6), PID: 1794248, Thread 1794399: 
2025-07-17T11:07:25.8425847Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8426053Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8426323Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8426522Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8427019Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8427227Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8427440Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8427447Z 
2025-07-17T11:07:25.8427538Z SIGABRT(6), PID: 1794253, Thread 1794700: 
2025-07-17T11:07:25.8428004Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8428213Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8428415Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8428615Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8429108Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8429321Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8429530Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8429535Z 
2025-07-17T11:07:25.8429630Z SIGABRT(6), PID: 1794248, Thread 1794401: 
2025-07-17T11:07:25.8430091Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8430299Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8430502Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8430703Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8431189Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8431401Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8431682Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8431686Z 
2025-07-17T11:07:25.8431778Z SIGABRT(6), PID: 1794253, Thread 1794703: 
2025-07-17T11:07:25.8432242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8432449Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8432714Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8432918Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8433472Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8433688Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8433896Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8433901Z 
2025-07-17T11:07:25.8434057Z SIGABRT(6), PID: 1794248, Thread 1794403: 
2025-07-17T11:07:25.8434523Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8434733Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8434936Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8435136Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8435633Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8435847Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8436054Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8436060Z 
2025-07-17T11:07:25.8436154Z SIGABRT(6), PID: 1794253, Thread 1794706: 
2025-07-17T11:07:25.8436623Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8436832Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8437035Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8437235Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8437726Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8437936Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8438144Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8438152Z 
2025-07-17T11:07:25.8438240Z SIGABRT(6), PID: 1794248, Thread 1794405: 
2025-07-17T11:07:25.8438714Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8438917Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8439123Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8439321Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8439886Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8440094Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8440307Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8440311Z 
2025-07-17T11:07:25.8440400Z SIGABRT(6), PID: 1794253, Thread 1794709: 
2025-07-17T11:07:25.8440928Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8441137Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8441404Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8441606Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8442104Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8442375Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8442587Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8442591Z 
2025-07-17T11:07:25.8442682Z SIGABRT(6), PID: 1794248, Thread 1794407: 
2025-07-17T11:07:25.8443157Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8443363Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8443568Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8443764Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8444258Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8444468Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8444680Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8444685Z 
2025-07-17T11:07:25.8444774Z SIGABRT(6), PID: 1794253, Thread 1794712: 
2025-07-17T11:07:25.8445242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8445446Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8445654Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8445849Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8446342Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8446548Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8446761Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8446765Z 
2025-07-17T11:07:25.8446854Z SIGABRT(6), PID: 1794253, Thread 1794715: 
2025-07-17T11:07:25.8447324Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8447530Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8447823Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8448022Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8448519Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8448789Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8449002Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8449095Z SIGABRT(6), PID: 1794248, Thread 1794409: 
2025-07-17T11:07:25.8449620Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8449833Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8450041Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8450304Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8450802Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8451012Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8451218Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8451223Z 
2025-07-17T11:07:25.8451226Z 
2025-07-17T11:07:25.8451320Z SIGABRT(6), PID: 1794253, Thread 1794717: 
2025-07-17T11:07:25.8451783Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8451993Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8452193Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8452394Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8452890Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8453098Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8453305Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8453310Z 
2025-07-17T11:07:25.8453401Z SIGABRT(6), PID: 1794248, Thread 1794411: 
2025-07-17T11:07:25.8453866Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8454071Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8454275Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8454473Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8454965Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8455179Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8455386Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8455390Z 
2025-07-17T11:07:25.8455554Z SIGABRT(6), PID: 1794253, Thread 1794719: 
2025-07-17T11:07:25.8456088Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8456300Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8456504Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8456785Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8457283Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8457562Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8457773Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8457780Z 
2025-07-17T11:07:25.8457875Z SIGABRT(6), PID: 1794248, Thread 1794413: 
2025-07-17T11:07:25.8458345Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8458629Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8458837Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8459043Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8459536Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8459748Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8459965Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8459969Z 
2025-07-17T11:07:25.8460059Z SIGABRT(6), PID: 1794253, Thread 1794721: 
2025-07-17T11:07:25.8460527Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8460734Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8460938Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8461133Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8461626Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8461831Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8462040Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8462044Z 
2025-07-17T11:07:25.8462133Z SIGABRT(6), PID: 1794248, Thread 1794415: 
2025-07-17T11:07:25.8462607Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8462812Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8463016Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8463212Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8463707Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8464001Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8464216Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8464221Z 
2025-07-17T11:07:25.8464313Z SIGABRT(6), PID: 1794253, Thread 1794723: 
2025-07-17T11:07:25.8464844Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8465054Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8465259Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8465455Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8466011Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8466226Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8466437Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8466498Z 
2025-07-17T11:07:25.8466591Z SIGABRT(6), PID: 1794248, Thread 1794417: 
2025-07-17T11:07:25.8467060Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8467268Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8467472Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8467669Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8468166Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8468371Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8468582Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8468586Z 
2025-07-17T11:07:25.8468674Z SIGABRT(6), PID: 1794253, Thread 1794725: 
2025-07-17T11:07:25.8469151Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8469363Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8469567Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8469768Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8470262Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8470473Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8470678Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8470682Z 
2025-07-17T11:07:25.8470774Z SIGABRT(6), PID: 1794248, Thread 1794418: 
2025-07-17T11:07:25.8471239Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8471448Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8471653Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8471925Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8472417Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8472631Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8472898Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8472903Z 
2025-07-17T11:07:25.8472996Z SIGABRT(6), PID: 1794253, Thread 1794727: 
2025-07-17T11:07:25.8473462Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8473736Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8473945Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8474145Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8474639Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8474933Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8475145Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8475149Z 
2025-07-17T11:07:25.8475242Z SIGABRT(6), PID: 1794248, Thread 1794419: 
2025-07-17T11:07:25.8475710Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8475917Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8476121Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8476319Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8476814Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8477026Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8477233Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8477237Z 
2025-07-17T11:07:25.8477329Z SIGABRT(6), PID: 1794253, Thread 1794729: 
2025-07-17T11:07:25.8477795Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8478005Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8478207Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8478410Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8478908Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8479117Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8479321Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8479331Z 
2025-07-17T11:07:25.8479420Z SIGABRT(6), PID: 1794248, Thread 1794420: 
2025-07-17T11:07:25.8479892Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8480170Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8480376Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8480576Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8481135Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8481347Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8481557Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8481561Z 
2025-07-17T11:07:25.8481711Z SIGABRT(6), PID: 1794253, Thread 1794731: 
2025-07-17T11:07:25.8482201Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8482416Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8482697Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8482898Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8483406Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8483617Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8483835Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8483839Z 
2025-07-17T11:07:25.8483931Z SIGABRT(6), PID: 1794248, Thread 1794421: 
2025-07-17T11:07:25.8484401Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8484610Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8484816Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8485016Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8485518Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8485723Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8485937Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8485943Z 
2025-07-17T11:07:25.8486032Z SIGABRT(6), PID: 1794253, Thread 1794733: 
2025-07-17T11:07:25.8486501Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8486707Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8486912Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8487110Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8487607Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8487816Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8488101Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8488105Z 
2025-07-17T11:07:25.8488193Z SIGABRT(6), PID: 1794248, Thread 1794422: 
2025-07-17T11:07:25.8488670Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8488877Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8489145Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8489345Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8489898Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8490110Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8490325Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8490329Z 
2025-07-17T11:07:25.8490416Z SIGABRT(6), PID: 1794253, Thread 1794734: 
2025-07-17T11:07:25.8491063Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8491274Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8491476Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8491676Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8492172Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8492383Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8492590Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8492595Z 
2025-07-17T11:07:25.8492688Z SIGABRT(6), PID: 1794248, Thread 1794423: 
2025-07-17T11:07:25.8493155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8493367Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8493570Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8493770Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8494263Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8494479Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8494687Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8494692Z 
2025-07-17T11:07:25.8494785Z SIGABRT(6), PID: 1794253, Thread 1794736: 
2025-07-17T11:07:25.8495248Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8495457Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8495658Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8495858Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8496406Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8496701Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8496911Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8496915Z 
2025-07-17T11:07:25.8497008Z SIGABRT(6), PID: 1794253, Thread 1794738: 
2025-07-17T11:07:25.8497543Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8497759Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8498033Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8498235Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8498732Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8499019Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8499229Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8499233Z 
2025-07-17T11:07:25.8499327Z SIGABRT(6), PID: 1794248, Thread 1794424: 
2025-07-17T11:07:25.8499803Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8500013Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8500218Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8500421Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8500919Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8501133Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8501343Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8501435Z SIGABRT(6), PID: 1794253, Thread 1794740: 
2025-07-17T11:07:25.8501904Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8502113Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8502320Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8502516Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8503008Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8503218Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8503430Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8503434Z 
2025-07-17T11:07:25.8503437Z 
2025-07-17T11:07:25.8503525Z SIGABRT(6), PID: 1794253, Thread 1794742: 
2025-07-17T11:07:25.8504002Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8504206Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8504481Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8504677Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8505174Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8505456Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8505671Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8505675Z 
2025-07-17T11:07:25.8505764Z SIGABRT(6), PID: 1794248, Thread 1794425: 
2025-07-17T11:07:25.8506321Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8506534Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8506739Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8506936Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8507498Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8507708Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8507919Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8507924Z 
2025-07-17T11:07:25.8508012Z SIGABRT(6), PID: 1794253, Thread 1794744: 
2025-07-17T11:07:25.8508483Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8508696Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8508900Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8509096Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8509592Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8509796Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8510006Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8510099Z SIGABRT(6), PID: 1794248, Thread 1794426: 
2025-07-17T11:07:25.8510573Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8510777Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8510985Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8511183Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8511675Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8511882Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8512094Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8512100Z 
2025-07-17T11:07:25.8512103Z 
2025-07-17T11:07:25.8512196Z SIGABRT(6), PID: 1794248, Thread 1794427: 
2025-07-17T11:07:25.8512733Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8512942Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8513147Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8513410Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8513911Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8514121Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8514388Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8514484Z SIGABRT(6), PID: 1794253, Thread 1794746: 
2025-07-17T11:07:25.8514950Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8515219Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8515420Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8515621Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8516110Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8516319Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8516527Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8516533Z 
2025-07-17T11:07:25.8516537Z 
2025-07-17T11:07:25.8516627Z SIGABRT(6), PID: 1794248, Thread 1794428: 
2025-07-17T11:07:25.8517094Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8517304Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8517507Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8517708Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8518206Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8518417Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8518626Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8518718Z SIGABRT(6), PID: 1794253, Thread 1794748: 
2025-07-17T11:07:25.8519182Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8519389Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8519593Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8519794Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8520287Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8520497Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8520775Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8520779Z 
2025-07-17T11:07:25.8520782Z 
2025-07-17T11:07:25.8520875Z SIGABRT(6), PID: 1794248, Thread 1794429: 
2025-07-17T11:07:25.8521342Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8521611Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8521818Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8522020Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8522568Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8522782Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8522991Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8523141Z SIGABRT(6), PID: 1794253, Thread 1794750: 
2025-07-17T11:07:25.8523610Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8523816Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8524019Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8524221Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8524716Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8524922Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8525134Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8525140Z 
2025-07-17T11:07:25.8525144Z 
2025-07-17T11:07:25.8525233Z SIGABRT(6), PID: 1794253, Thread 1794752: 
2025-07-17T11:07:25.8525704Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8525909Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8526113Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8526310Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8526805Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8527010Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8527222Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8527225Z 
2025-07-17T11:07:25.8527315Z SIGABRT(6), PID: 1794248, Thread 1794430: 
2025-07-17T11:07:25.8527792Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8527996Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8528204Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8528477Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8528974Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8529184Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8529396Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8529463Z 
2025-07-17T11:07:25.8529556Z SIGABRT(6), PID: 1794253, Thread 1794754: 
2025-07-17T11:07:25.8530028Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8530295Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8530504Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8530706Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8531207Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8531476Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8531691Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8531695Z 
2025-07-17T11:07:25.8531785Z SIGABRT(6), PID: 1794248, Thread 1794432: 
2025-07-17T11:07:25.8532253Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8532460Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8532670Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8532867Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8533369Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8533582Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8533790Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8533794Z 
2025-07-17T11:07:25.8533885Z SIGABRT(6), PID: 1794253, Thread 1794756: 
2025-07-17T11:07:25.8534357Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8534568Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8534773Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8534971Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8535464Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8535674Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8535882Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8535886Z 
2025-07-17T11:07:25.8536040Z SIGABRT(6), PID: 1794248, Thread 1794434: 
2025-07-17T11:07:25.8536506Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8536806Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8537011Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8537214Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8537794Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8538011Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8538220Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8538224Z 
2025-07-17T11:07:25.8538390Z SIGABRT(6), PID: 1794253, Thread 1794757: 
2025-07-17T11:07:25.8538859Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8539070Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8539271Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8539546Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8540037Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8540245Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8540453Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8540459Z 
2025-07-17T11:07:25.8540551Z SIGABRT(6), PID: 1794248, Thread 1794436: 
2025-07-17T11:07:25.8541019Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8541227Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8541430Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8541632Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8542123Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8542335Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8542545Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8542551Z 
2025-07-17T11:07:25.8542645Z SIGABRT(6), PID: 1794253, Thread 1794758: 
2025-07-17T11:07:25.8543112Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8543323Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8543525Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8543726Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8544228Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8544435Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8544647Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8544722Z 
2025-07-17T11:07:25.8544814Z SIGABRT(6), PID: 1794248, Thread 1794438: 
2025-07-17T11:07:25.8545287Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8545492Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8545759Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8545960Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8546456Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8546721Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8546939Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8546943Z 
2025-07-17T11:07:25.8547033Z SIGABRT(6), PID: 1794253, Thread 1794759: 
2025-07-17T11:07:25.8547499Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8547767Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8547974Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8548170Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8548667Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8548872Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8549087Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8549091Z 
2025-07-17T11:07:25.8549179Z SIGABRT(6), PID: 1794248, Thread 1794440: 
2025-07-17T11:07:25.8549649Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8549855Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8550061Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8550258Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8550757Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8550966Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8551176Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8551182Z 
2025-07-17T11:07:25.8551269Z SIGABRT(6), PID: 1794253, Thread 1794760: 
2025-07-17T11:07:25.8551742Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8551946Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8552151Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8552347Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8552838Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8553116Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8553328Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8553335Z 
2025-07-17T11:07:25.8553424Z SIGABRT(6), PID: 1794248, Thread 1794442: 
2025-07-17T11:07:25.8553955Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8554171Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8554373Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8554632Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8555125Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8555335Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8555606Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8555610Z 
2025-07-17T11:07:25.8555701Z SIGABRT(6), PID: 1794253, Thread 1794761: 
2025-07-17T11:07:25.8556172Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8556378Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8556582Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8556781Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8557275Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8557487Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8557695Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8557698Z 
2025-07-17T11:07:25.8557792Z SIGABRT(6), PID: 1794248, Thread 1794444: 
2025-07-17T11:07:25.8558257Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8558465Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8558668Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8558868Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8559359Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8559571Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8559781Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8559785Z 
2025-07-17T11:07:25.8559877Z SIGABRT(6), PID: 1794253, Thread 1794762: 
2025-07-17T11:07:25.8560345Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8560556Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8560828Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8561030Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8561519Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8561732Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8561998Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8562003Z 
2025-07-17T11:07:25.8562098Z SIGABRT(6), PID: 1794248, Thread 1794446: 
2025-07-17T11:07:25.8562630Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8562842Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8563048Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8563247Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8563800Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8564022Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8564232Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8564240Z 
2025-07-17T11:07:25.8564327Z SIGABRT(6), PID: 1794253, Thread 1794763: 
2025-07-17T11:07:25.8564797Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8565006Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8565212Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8565411Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8565906Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8566112Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8566323Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8566327Z 
2025-07-17T11:07:25.8566416Z SIGABRT(6), PID: 1794248, Thread 1794449: 
2025-07-17T11:07:25.8566884Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8567089Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8567294Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8567494Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8567990Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8568196Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8568411Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8568415Z 
2025-07-17T11:07:25.8568506Z SIGABRT(6), PID: 1794253, Thread 1794764: 
2025-07-17T11:07:25.8569064Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8569271Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8569474Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8569670Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8570233Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8570443Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8570715Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8570719Z 
2025-07-17T11:07:25.8570812Z SIGABRT(6), PID: 1794248, Thread 1794452: 
2025-07-17T11:07:25.8571284Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8571546Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8571751Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8571949Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8572441Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8572649Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8572862Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8572869Z 
2025-07-17T11:07:25.8572957Z SIGABRT(6), PID: 1794253, Thread 1794765: 
2025-07-17T11:07:25.8573429Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8573637Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8573845Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8574041Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8574544Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8574755Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8574967Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8574971Z 
2025-07-17T11:07:25.8575060Z SIGABRT(6), PID: 1794248, Thread 1794455: 
2025-07-17T11:07:25.8575529Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8575740Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8576002Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8576206Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8576699Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8576909Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8577203Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8577295Z SIGABRT(6), PID: 1794253, Thread 1794766: 
2025-07-17T11:07:25.8577766Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8578048Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8578254Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8578454Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8579173Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8579649Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8579864Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8579869Z 
2025-07-17T11:07:25.8579961Z 
2025-07-17T11:07:25.8580059Z SIGABRT(6), PID: 1794248, Thread 1794458: 
2025-07-17T11:07:25.8580536Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8580749Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8580953Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8581152Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8581649Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8581865Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8582072Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8582169Z SIGABRT(6), PID: 1794253, Thread 1794767: 
2025-07-17T11:07:25.8582637Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8582845Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8583045Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8583249Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8583738Z frame #4: <unknown function> + 0x3475ab (0x7f99e26915ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8583949Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8584159Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8584162Z 
2025-07-17T11:07:25.8584167Z 
2025-07-17T11:07:25.8584259Z SIGABRT(6), PID: 1794253, Thread 1795316: 
2025-07-17T11:07:25.8584725Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8585182Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f98f3ef008a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8585390Z frame #2: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8585678Z frame #3: pthread_kill + 0x12c (0x7f99e57fb9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8585846Z frame #4: raise + 0x16 (0x7f99e57a7476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8586007Z frame #5: abort + 0xd3 (0x7f99e578d7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8586247Z frame #6: <unknown function> + 0x972cc (0x7f9861a792cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8586478Z frame #7: <unknown function> + 0x95f47 (0x7f9861a77f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8586763Z frame #8: <unknown function> + 0x34b71 (0x7f9861a16b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8586979Z frame #9: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8587192Z frame #10: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8587258Z 
2025-07-17T11:07:25.8587356Z SIGABRT(6), PID: 1794248, Thread 1794461: 
2025-07-17T11:07:25.8587829Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8588041Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8588309Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8588510Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8589012Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8589222Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8589437Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8589529Z SIGABRT(6), PID: 1794253, Thread 1798234: 
2025-07-17T11:07:25.8589998Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8590203Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8590373Z frame #2: __poll + 0x4f (0x7f99e587dbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8590781Z frame #3: <unknown function> + 0x489f (0x7f99e538989f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.8591020Z frame #4: <unknown function> + 0x1483bc (0x55a1b141a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8591271Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8591503Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8591742Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8591968Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8592215Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55a1b1402a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8592448Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8592699Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a1b1400c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8592927Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8593164Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8593390Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8593695Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8593915Z frame #16: <unknown function> + 0x147734 (0x55a1b1419734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8594147Z frame #17: <unknown function> + 0x21ca82 (0x55a1b14eea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8594367Z frame #18: <unknown function> + 0x21c9e4 (0x55a1b14ee9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8594641Z frame #19: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8594868Z frame #20: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8594872Z 
2025-07-17T11:07:25.8594876Z 
2025-07-17T11:07:25.8594970Z SIGABRT(6), PID: 1794248, Thread 1794464: 
2025-07-17T11:07:25.8595509Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8595729Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8595939Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8596222Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8596731Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8596944Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8597159Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8597163Z 
2025-07-17T11:07:25.8597256Z SIGABRT(6), PID: 1794248, Thread 1794467: 
2025-07-17T11:07:25.8597731Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8597942Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8598150Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8598350Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8598851Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8599058Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8599270Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8599275Z 
2025-07-17T11:07:25.8599367Z SIGABRT(6), PID: 1794253, Thread 1798235: 
2025-07-17T11:07:25.8599838Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8600044Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8600250Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8600472Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f99e57f92dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8600928Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f991fc38048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.8601355Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f991fc3abf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.8601595Z frame #6: <unknown function> + 0xdc253 (0x7f98f215c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8601885Z frame #7: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8602102Z frame #8: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8602107Z 
2025-07-17T11:07:25.8602198Z SIGABRT(6), PID: 1794248, Thread 1794471: 
2025-07-17T11:07:25.8602736Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8602954Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8603159Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8603360Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8603926Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8604150Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8604361Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8604425Z 
2025-07-17T11:07:25.8604521Z SIGABRT(6), PID: 1794248, Thread 1794474: 
2025-07-17T11:07:25.8604997Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8605206Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8605410Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8605613Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8606112Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8606320Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8611708Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8611845Z SIGABRT(6), PID: 1794253, Thread 1798236: 
2025-07-17T11:07:25.8612364Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8612595Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8612803Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8613038Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f99e57f92dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8613529Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f991fc32741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.8613780Z frame #5: <unknown function> + 0xdc253 (0x7f98f215c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8614002Z frame #6: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8614222Z frame #7: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8614228Z 
2025-07-17T11:07:25.8614232Z 
2025-07-17T11:07:25.8614327Z SIGABRT(6), PID: 1794253, Thread 1798343: 
2025-07-17T11:07:25.8614826Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8615040Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8615336Z frame #2: ioctl + 0x3f (0x7f99e587f94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8615579Z frame #3: <unknown function> + 0x13eef0 (0x7f9861b20ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8615814Z frame #4: <unknown function> + 0x1378c4 (0x7f9861b198c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8616113Z frame #5: <unknown function> + 0x95b2b (0x7f9861a77b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8616426Z frame #6: <unknown function> + 0x34b71 (0x7f9861a16b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8616646Z frame #7: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8616862Z frame #8: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8617030Z SIGABRT(6), PID: 1794248, Thread 1794477: 
2025-07-17T11:07:25.8617511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8617718Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8617925Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8618203Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8618739Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8618955Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8619171Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8619177Z 
2025-07-17T11:07:25.8619181Z 
2025-07-17T11:07:25.8619274Z SIGABRT(6), PID: 1794248, Thread 1794480: 
2025-07-17T11:07:25.8619762Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8619970Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8620180Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8620386Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8620893Z frame #4: <unknown function> + 0x3475ab (0x7f7ea02625ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8621109Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8621327Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8621333Z 
2025-07-17T11:07:25.8621425Z SIGABRT(6), PID: 1794253, Thread 1799061: 
2025-07-17T11:07:25.8621905Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8622119Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8622289Z frame #2: __poll + 0x4f (0x7f99e587dbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8622506Z frame #3: <unknown function> + 0xd4540 (0x7f98a9c90540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8622711Z frame #4: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8622923Z frame #5: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8622927Z 
2025-07-17T11:07:25.8623019Z SIGABRT(6), PID: 1794248, Thread 1795304: 
2025-07-17T11:07:25.8623495Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8624029Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f7db1ac208a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8624249Z frame #2: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8624505Z frame #3: pthread_kill + 0x12c (0x7f7ea33cc9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8624677Z frame #4: raise + 0x16 (0x7f7ea3378476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8624838Z frame #5: abort + 0xd3 (0x7f7ea335e7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8625082Z frame #6: <unknown function> + 0x972cc (0x7f7d1f64b2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8625484Z frame #7: <unknown function> + 0x95f47 (0x7f7d1f649f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8625715Z frame #8: <unknown function> + 0x34b71 (0x7f7d1f5e8b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8625927Z frame #9: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8626206Z frame #10: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8626212Z 
2025-07-17T11:07:25.8626304Z SIGABRT(6), PID: 1794253, Thread 1799068: 
2025-07-17T11:07:25.8626777Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8626986Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8627157Z frame #2: __poll + 0x4f (0x7f99e587dbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8627374Z frame #3: <unknown function> + 0xd65a2 (0x7f98a9c925a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8627585Z frame #4: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8627798Z frame #5: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8627804Z 
2025-07-17T11:07:25.8627899Z SIGABRT(6), PID: 1794248, Thread 1798637: 
2025-07-17T11:07:25.8628361Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8628570Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8628734Z frame #2: __poll + 0x4f (0x7f7ea344ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8629143Z frame #3: <unknown function> + 0x489f (0x7f7ea2f5a89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.8629374Z frame #4: <unknown function> + 0x1483bc (0x55dbf51353bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8629623Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8629849Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8630086Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8630306Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8630555Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55dbf511da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8630784Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8631037Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55dbf511bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8631266Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8631578Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8631804Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8632043Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8632264Z frame #16: <unknown function> + 0x147734 (0x55dbf5134734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8632552Z frame #17: <unknown function> + 0x21ca82 (0x55dbf5209a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8632777Z frame #18: <unknown function> + 0x21c9e4 (0x55dbf52099e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8632994Z frame #19: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8633290Z frame #20: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8633298Z 
2025-07-17T11:07:25.8633395Z SIGABRT(6), PID: 1794253, Thread 1799079: 
2025-07-17T11:07:25.8633874Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8634162Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8634370Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8634580Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8634796Z frame #4: <unknown function> + 0xd1e7f (0x7f98a9c8de7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8635005Z frame #5: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8635213Z frame #6: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8635220Z 
2025-07-17T11:07:25.8635313Z SIGABRT(6), PID: 1794253, Thread 1799080: 
2025-07-17T11:07:25.8635779Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8635988Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8636193Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8636401Z frame #3: <unknown function> + 0x9ca53 (0x7f99e5801a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8636654Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55a1b13e6bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8636885Z frame #5: <unknown function> + 0x1d314f (0x55a1b14a514f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8637108Z frame #6: <unknown function> + 0x1d2f6b (0x55a1b14a4f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8637328Z frame #7: <unknown function> + 0x6df36 (0x55a1b133ff36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8637566Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8637792Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8638029Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8638258Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8638493Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8638717Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8638954Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8639249Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8639482Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8639702Z frame #17: <unknown function> + 0x147734 (0x55a1b1419734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8639925Z frame #18: <unknown function> + 0x21ca82 (0x55a1b14eea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8640208Z frame #19: <unknown function> + 0x21c9e4 (0x55a1b14ee9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8640425Z frame #20: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8640639Z frame #21: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8640644Z 
2025-07-17T11:07:25.8640790Z SIGABRT(6), PID: 1794253, Thread 1799089: 
2025-07-17T11:07:25.8641266Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8641477Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8641644Z frame #2: read + 0x4c (0x7f99e587981c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8641891Z frame #3: _Py_read + 0x4c (0x55a1b14aca6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8642119Z frame #4: <unknown function> + 0x208c3b (0x55a1b14dac3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8642338Z frame #5: <unknown function> + 0x14648d (0x55a1b141848d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8642583Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55a1b1427215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8642805Z frame #7: <unknown function> + 0x208ae7 (0x55a1b14daae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8643024Z frame #8: <unknown function> + 0x208472 (0x55a1b14da472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8643245Z frame #9: <unknown function> + 0x1da6ee (0x55a1b14ac6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8643469Z frame #10: <unknown function> + 0x1483bc (0x55a1b141a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8643707Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8643932Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8644175Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55a1b13fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8644399Z frame #14: <unknown function> + 0x147734 (0x55a1b1419734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8644648Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a1b1400c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8644872Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8645109Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8645332Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55a1b140e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8645565Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55a1b13fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8645784Z frame #20: <unknown function> + 0x147734 (0x55a1b1419734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8646003Z frame #21: <unknown function> + 0x21ca82 (0x55a1b14eea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8646226Z frame #22: <unknown function> + 0x21c9e4 (0x55a1b14ee9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8646435Z frame #23: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8646648Z frame #24: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8646720Z 
2025-07-17T11:07:25.8646813Z SIGABRT(6), PID: 1794248, Thread 1798639: 
2025-07-17T11:07:25.8647289Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8647501Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8647709Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8647995Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f7ea33ca2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8648447Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f7ddd80a048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.8648933Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f7ddd80cbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.8649173Z frame #6: <unknown function> + 0xdc253 (0x7f7dafd2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8649383Z frame #7: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8649654Z frame #8: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8649658Z 
2025-07-17T11:07:25.8649749Z SIGABRT(6), PID: 1794248, Thread 1798640: 
2025-07-17T11:07:25.8650224Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8650433Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8650636Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8650862Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f7ea33ca2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8651337Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f7ddd804741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.8651571Z frame #5: <unknown function> + 0xdc253 (0x7f7dafd2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8651780Z frame #6: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8651994Z frame #7: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8651998Z 
2025-07-17T11:07:25.8652088Z SIGABRT(6), PID: 1794248, Thread 1798726: 
2025-07-17T11:07:25.8652565Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8652774Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8652944Z frame #2: ioctl + 0x3f (0x7f7ea345094f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8653182Z frame #3: <unknown function> + 0x13eef0 (0x7f7d1f6f2ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8653416Z frame #4: <unknown function> + 0x1378c4 (0x7f7d1f6eb8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8653644Z frame #5: <unknown function> + 0x95b2b (0x7f7d1f649b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8653874Z frame #6: <unknown function> + 0x34b71 (0x7f7d1f5e8b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8654082Z frame #7: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8654292Z frame #8: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8654297Z 
2025-07-17T11:07:25.8654385Z SIGABRT(6), PID: 1794248, Thread 1799058: 
2025-07-17T11:07:25.8654857Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8655141Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8655311Z frame #2: __poll + 0x4f (0x7f7ea344ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8655521Z frame #3: <unknown function> + 0xd4540 (0x7f7d67862540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8655729Z frame #4: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8656117Z frame #5: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8656123Z 
2025-07-17T11:07:25.8656227Z SIGABRT(6), PID: 1794248, Thread 1799066: 
2025-07-17T11:07:25.8656792Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8657012Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8657180Z frame #2: __poll + 0x4f (0x7f7ea344ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8657395Z frame #3: <unknown function> + 0xd65a2 (0x7f7d678645a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8657678Z frame #4: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8657888Z frame #5: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8657894Z 
2025-07-17T11:07:25.8657983Z SIGABRT(6), PID: 1794253, Thread 1799104: 
2025-07-17T11:07:25.8658455Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8658665Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8658871Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8659077Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8659459Z frame #4: <unknown function> + 0x5a02023 (0x7f9929c26023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8660030Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9929c2a1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8660633Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9929c211d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8661018Z frame #7: <unknown function> + 0x79d655 (0x7f993b111655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8661248Z frame #8: <unknown function> + 0xdc253 (0x7f98f215c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8661458Z frame #9: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8661676Z frame #10: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8661683Z 
2025-07-17T11:07:25.8661773Z SIGABRT(6), PID: 1794248, Thread 1799074: 
2025-07-17T11:07:25.8662250Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8662459Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8662661Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8662866Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8663082Z frame #4: <unknown function> + 0xd1e7f (0x7f7d6785fe7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.8663375Z frame #5: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8663583Z frame #6: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8663590Z 
2025-07-17T11:07:25.8663682Z SIGABRT(6), PID: 1794248, Thread 1799085: 
2025-07-17T11:07:25.8664233Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8664444Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8664646Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8664915Z frame #3: <unknown function> + 0x9ca53 (0x7f7ea33d2a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8665178Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55dbf5101bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8665412Z frame #5: <unknown function> + 0x1d314f (0x55dbf51c014f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8665635Z frame #6: <unknown function> + 0x1d2f6b (0x55dbf51bff6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8665945Z frame #7: <unknown function> + 0x6df36 (0x55dbf505af36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8666191Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8666418Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8666657Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8666889Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8667122Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8667351Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8667585Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8667811Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8668042Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8668266Z frame #17: <unknown function> + 0x147734 (0x55dbf5134734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8668486Z frame #18: <unknown function> + 0x21ca82 (0x55dbf5209a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8668710Z frame #19: <unknown function> + 0x21c9e4 (0x55dbf52099e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8668923Z frame #20: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8669140Z frame #21: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8669144Z 
2025-07-17T11:07:25.8669233Z SIGABRT(6), PID: 1794248, Thread 1799097: 
2025-07-17T11:07:25.8669706Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8669918Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8670084Z frame #2: read + 0x4c (0x7f7ea344a81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8670265Z frame #3: _Py_read + 0x4c (0x55dbf51c7a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8670495Z frame #4: <unknown function> + 0x208c3b (0x55dbf51f5c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8670716Z frame #5: <unknown function> + 0x14648d (0x55dbf513348d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8671034Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55dbf5142215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8671253Z frame #7: <unknown function> + 0x208ae7 (0x55dbf51f5ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8671478Z frame #8: <unknown function> + 0x208472 (0x55dbf51f5472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8671699Z frame #9: <unknown function> + 0x1da6ee (0x55dbf51c76ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8671981Z frame #10: <unknown function> + 0x1483bc (0x55dbf51353bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8672226Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8672452Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8672749Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55dbf51193bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8672977Z frame #14: <unknown function> + 0x147734 (0x55dbf5134734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8673227Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55dbf511bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8673519Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8673755Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8673982Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55dbf51290ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8674223Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55dbf51197b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8674443Z frame #20: <unknown function> + 0x147734 (0x55dbf5134734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8674666Z frame #21: <unknown function> + 0x21ca82 (0x55dbf5209a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8674887Z frame #22: <unknown function> + 0x21c9e4 (0x55dbf52099e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8675099Z frame #23: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8675311Z frame #24: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8675317Z 
2025-07-17T11:07:25.8675407Z SIGABRT(6), PID: 1794253, Thread 1799105: 
2025-07-17T11:07:25.8675883Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8676091Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8676298Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8676499Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8676878Z frame #4: <unknown function> + 0x5a02023 (0x7f9929c26023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8677443Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9929c2a1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8678042Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9929c211d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8678425Z frame #7: <unknown function> + 0x79d655 (0x7f993b111655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8678654Z frame #8: <unknown function> + 0xdc253 (0x7f98f215c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8678862Z frame #9: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8679155Z frame #10: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8679160Z 
2025-07-17T11:07:25.8679252Z SIGABRT(6), PID: 1794248, Thread 1799152: 
2025-07-17T11:07:25.8679725Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8679991Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8680204Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8680406Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8680837Z frame #4: <unknown function> + 0x5a02023 (0x7f7de77f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8681401Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7de77fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8681996Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7de77f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8682451Z frame #7: <unknown function> + 0x79d655 (0x7f7df8ce3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8682687Z frame #8: <unknown function> + 0xdc253 (0x7f7dafd2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8682899Z frame #9: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8683120Z frame #10: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8683124Z 
2025-07-17T11:07:25.8683216Z SIGABRT(6), PID: 1794253, Thread 1799106: 
2025-07-17T11:07:25.8683719Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8683936Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8684144Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8684354Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8684730Z frame #4: <unknown function> + 0x5a02023 (0x7f9929c26023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8685306Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9929c2a1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8685909Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9929c211d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8686296Z frame #7: <unknown function> + 0x79d655 (0x7f993b111655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8686527Z frame #8: <unknown function> + 0xdc253 (0x7f98f215c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8686744Z frame #9: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8686962Z frame #10: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8686966Z 
2025-07-17T11:07:25.8687059Z SIGABRT(6), PID: 1794248, Thread 1799153: 
2025-07-17T11:07:25.8687538Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8687822Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8688028Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8688238Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8688608Z frame #4: <unknown function> + 0x5a02023 (0x7f7de77f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8689236Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7de77fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8689897Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7de77f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8690288Z frame #7: <unknown function> + 0x79d655 (0x7f7df8ce3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8690517Z frame #8: <unknown function> + 0xdc253 (0x7f7dafd2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8690794Z frame #9: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8691008Z frame #10: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8691014Z 
2025-07-17T11:07:25.8691109Z SIGABRT(6), PID: 1794253, Thread 1799107: 
2025-07-17T11:07:25.8691585Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8691798Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8692002Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8692209Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8692578Z frame #4: <unknown function> + 0x5a02023 (0x7f9929c26023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8693146Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9929c2a1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8693734Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9929c211d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8694111Z frame #7: <unknown function> + 0x79d655 (0x7f993b111655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8694342Z frame #8: <unknown function> + 0xdc253 (0x7f98f215c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8694558Z frame #9: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8694773Z frame #10: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8694779Z 
2025-07-17T11:07:25.8694869Z SIGABRT(6), PID: 1794248, Thread 1799154: 
2025-07-17T11:07:25.8695347Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8695555Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8695763Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8696034Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8696522Z frame #4: <unknown function> + 0x5a02023 (0x7f7de77f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8697081Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7de77fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8697754Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7de77f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8698138Z frame #7: <unknown function> + 0x79d655 (0x7f7df8ce3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8698370Z frame #8: <unknown function> + 0xdc253 (0x7f7dafd2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8698649Z frame #9: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8698873Z frame #10: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8698877Z 
2025-07-17T11:07:25.8698967Z SIGABRT(6), PID: 1794253, Thread 1799108: 
2025-07-17T11:07:25.8699509Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8699717Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8699925Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8700124Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8700494Z frame #4: <unknown function> + 0x5a02023 (0x7f9929c26023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8701050Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9929c2a1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8701642Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9929c211d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8702024Z frame #7: <unknown function> + 0x79d655 (0x7f993b111655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8702254Z frame #8: <unknown function> + 0xdc253 (0x7f98f215c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8702461Z frame #9: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8702675Z frame #10: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8702679Z 
2025-07-17T11:07:25.8702777Z SIGABRT(6), PID: 1794248, Thread 1799155: 
2025-07-17T11:07:25.8703251Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8703463Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8703669Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8703871Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8704237Z frame #4: <unknown function> + 0x5a02023 (0x7f7de77f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8704809Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7de77fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8705473Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7de77f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8705859Z frame #7: <unknown function> + 0x79d655 (0x7f7df8ce3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8706090Z frame #8: <unknown function> + 0xdc253 (0x7f7dafd2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8706362Z frame #9: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8706579Z frame #10: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8706583Z 
2025-07-17T11:07:25.8706675Z SIGABRT(6), PID: 1794253, Thread 1799109: 
2025-07-17T11:07:25.8707207Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8707428Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8707634Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8707915Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8708284Z frame #4: <unknown function> + 0x5a02023 (0x7f9929c26023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8708851Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9929c2a1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8709439Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9929c211d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8709822Z frame #7: <unknown function> + 0x79d655 (0x7f993b111655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8710052Z frame #8: <unknown function> + 0xdc253 (0x7f98f215c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8710264Z frame #9: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8710476Z frame #10: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8710486Z 
2025-07-17T11:07:25.8710577Z SIGABRT(6), PID: 1794248, Thread 1799156: 
2025-07-17T11:07:25.8711053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8711262Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8711469Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8711671Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8712046Z frame #4: <unknown function> + 0x5a02023 (0x7f7de77f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8712612Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7de77fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8713200Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7de77f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8713581Z frame #7: <unknown function> + 0x79d655 (0x7f7df8ce3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8713886Z frame #8: <unknown function> + 0xdc253 (0x7f7dafd2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8714100Z frame #9: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8714320Z frame #10: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8714326Z 
2025-07-17T11:07:25.8714418Z SIGABRT(6), PID: 1794253, Thread 1799110: 
2025-07-17T11:07:25.8714958Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8715169Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8715378Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8715638Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8716019Z frame #4: <unknown function> + 0x5a02023 (0x7f9929c26023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8716580Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9929c2a1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8717234Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9929c211d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8717612Z frame #7: <unknown function> + 0x79d655 (0x7f993b111655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8717849Z frame #8: <unknown function> + 0xdc253 (0x7f98f215c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8718056Z frame #9: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8718273Z frame #10: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8718277Z 
2025-07-17T11:07:25.8718369Z SIGABRT(6), PID: 1794248, Thread 1799157: 
2025-07-17T11:07:25.8718842Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8719052Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8719264Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8719460Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8719833Z frame #4: <unknown function> + 0x5a02023 (0x7f7de77f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8720396Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7de77fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8720992Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7de77f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8721376Z frame #7: <unknown function> + 0x79d655 (0x7f7df8ce3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8721606Z frame #8: <unknown function> + 0xdc253 (0x7f7dafd2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8721817Z frame #9: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8722031Z frame #10: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8722037Z 
2025-07-17T11:07:25.8722131Z SIGABRT(6), PID: 1794253, Thread 1799111: 
2025-07-17T11:07:25.8722673Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f98f3eefb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8722882Z frame #1: <unknown function> + 0x42520 (0x7f99e57a7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8723087Z frame #2: <unknown function> + 0x91117 (0x7f99e57f6117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8723349Z frame #3: pthread_cond_wait + 0x211 (0x7f99e57f8a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8723721Z frame #4: <unknown function> + 0x5a02023 (0x7f9929c26023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8724341Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9929c2a1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8724933Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9929c211d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8725317Z frame #7: <unknown function> + 0x79d655 (0x7f993b111655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8725627Z frame #8: <unknown function> + 0xdc253 (0x7f98f215c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8725846Z frame #9: <unknown function> + 0x94ac3 (0x7f99e57f9ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8726058Z frame #10: <unknown function> + 0x126850 (0x7f99e588b850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8726062Z 
2025-07-17T11:07:25.8726159Z SIGABRT(6), PID: 1794248, Thread 1799158: 
2025-07-17T11:07:25.8726632Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8726848Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8727053Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8727257Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8727627Z frame #4: <unknown function> + 0x5a02023 (0x7f7de77f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8728193Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7de77fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8728783Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7de77f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8729171Z frame #7: <unknown function> + 0x79d655 (0x7f7df8ce3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8729403Z frame #8: <unknown function> + 0xdc253 (0x7f7dafd2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8729614Z frame #9: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8729834Z frame #10: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8729839Z 
2025-07-17T11:07:25.8729929Z SIGABRT(6), PID: 1794248, Thread 1799159: 
2025-07-17T11:07:25.8730402Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7db1ac1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8730608Z frame #1: <unknown function> + 0x42520 (0x7f7ea3378520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8730816Z frame #2: <unknown function> + 0x91117 (0x7f7ea33c7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8731089Z frame #3: pthread_cond_wait + 0x211 (0x7f7ea33c9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8731459Z frame #4: <unknown function> + 0x5a02023 (0x7f7de77f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8732078Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7de77fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8732677Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7de77f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.8733116Z frame #7: <unknown function> + 0x79d655 (0x7f7df8ce3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8733364Z frame #8: <unknown function> + 0xdc253 (0x7f7dafd2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.8733575Z frame #9: <unknown function> + 0x94ac3 (0x7f7ea33caac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8733792Z frame #10: <unknown function> + 0x126850 (0x7f7ea345c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8733854Z 
2025-07-17T11:07:25.8733951Z SIGABRT(6), PID: 1794252, Thread 1794252: 
2025-07-17T11:07:25.8734434Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8734646Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8734891Z frame #2: <unknown function> + 0x7743f (0x7fe008eb243f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8735122Z frame #3: <unknown function> + 0x76c8e (0x7fe008eb1c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8735353Z frame #4: <unknown function> + 0x6ad69 (0x7fe008ea5d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8735570Z frame #5: <unknown function> + 0x3f54db (0x7fe099b0a4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8735784Z frame #6: <unknown function> + 0x3e11d8 (0x7fe099af61d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8736056Z frame #7: <unknown function> + 0xb2f86 (0x7fe0997c7f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8736267Z frame #8: <unknown function> + 0x8e501 (0x7fe0997a3501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8736649Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fe0c45aa7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.8737041Z frame #10: <unknown function> + 0xd307d9 (0x7fe0e2afd7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8737268Z frame #11: <unknown function> + 0x139902 (0x558910bb1902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8737527Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x558910ba8a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8737757Z frame #13: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8738006Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x558910ba8a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8738230Z frame #15: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8738476Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8738698Z frame #17: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8738934Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8739154Z frame #19: <unknown function> + 0x1474b2 (0x558910bbf4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8739447Z frame #20: PyObject_Call + 0xbc (0x558910bbfe4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8739668Z frame #21: <unknown function> + 0x1ff69d (0x558910c7769d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8739898Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x558910bad023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8740140Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x558910ba8cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8740440Z frame #24: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8740680Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x558910ba43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8740909Z frame #26: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8741217Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x558910ba43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8741445Z frame #28: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8741684Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x558910ba8a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8742019Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x558910bac4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8742241Z frame #31: _PyObject_Call_Prepend + 0x69 (0x558910bbdab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8742471Z frame #32: <unknown function> + 0x205949 (0x558910c7d949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8742696Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x558910bad023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8742940Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x558910ba8cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8743167Z frame #35: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8743406Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x558910ba43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8743627Z frame #37: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8743864Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x558910ba43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8744091Z frame #39: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8744332Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x558910ba43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8744552Z frame #41: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8744796Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x558910ba53f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8745020Z frame #43: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8745416Z frame #44: <unknown function> + 0x7ab3f7 (0x7fe0e25783f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8745635Z frame #45: <unknown function> + 0x13bc88 (0x558910bb3c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8745848Z frame #46: PyObject_Call + 0x20f (0x558910bbff9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8746090Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x558910ba998d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8746315Z frame #48: <unknown function> + 0x147608 (0x558910bbf608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8746555Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x558910ba6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8746784Z frame #50: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8747023Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x558910ba43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8747318Z frame #52: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8747560Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x558910ba53f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8747789Z frame #54: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8748026Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x558910ba43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8748314Z frame #56: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8748556Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x558910ba6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8748784Z frame #58: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8749103Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x558910ba6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8749336Z frame #60: <unknown function> + 0x147608 (0x558910bbf608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8749577Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x558910ba6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8749860Z frame #62: <unknown function> + 0x147608 (0x558910bbf608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8749865Z 
2025-07-17T11:07:25.8749960Z SIGABRT(6), PID: 1794252, Thread 1794593: 
2025-07-17T11:07:25.8750441Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8750658Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8750875Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8751079Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8751597Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8751806Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8752027Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8752031Z 
2025-07-17T11:07:25.8752124Z SIGABRT(6), PID: 1794252, Thread 1794596: 
2025-07-17T11:07:25.8752600Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8752811Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8753024Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8753225Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8753735Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8753949Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8754166Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8754170Z 
2025-07-17T11:07:25.8754263Z SIGABRT(6), PID: 1794252, Thread 1794600: 
2025-07-17T11:07:25.8754734Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8754949Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8755157Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8755427Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8755926Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8756141Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8756513Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8756518Z 
2025-07-17T11:07:25.8756619Z SIGABRT(6), PID: 1794252, Thread 1794602: 
2025-07-17T11:07:25.8757090Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8757387Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8757602Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8757805Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8758298Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8758575Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8758788Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8758792Z 
2025-07-17T11:07:25.8758888Z SIGABRT(6), PID: 1794252, Thread 1794605: 
2025-07-17T11:07:25.8759356Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8759571Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8759780Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8759980Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8760473Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8760688Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8760896Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8760900Z 
2025-07-17T11:07:25.8760995Z SIGABRT(6), PID: 1794252, Thread 1794608: 
2025-07-17T11:07:25.8761460Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8761677Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8761879Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8762081Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8762578Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8762794Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8763003Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8763008Z 
2025-07-17T11:07:25.8763110Z SIGABRT(6), PID: 1794252, Thread 1794612: 
2025-07-17T11:07:25.8763576Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8763869Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8764076Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8764290Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8764841Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8765059Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8765272Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8765339Z 
2025-07-17T11:07:25.8765432Z SIGABRT(6), PID: 1794252, Thread 1794614: 
2025-07-17T11:07:25.8765909Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8766118Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8766390Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8766588Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8767088Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8767300Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8767516Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8767520Z 
2025-07-17T11:07:25.8767612Z SIGABRT(6), PID: 1794252, Thread 1794616: 
2025-07-17T11:07:25.8768084Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8768291Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8768503Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8768704Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8769206Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8769415Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8769632Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8769638Z 
2025-07-17T11:07:25.8769728Z SIGABRT(6), PID: 1794252, Thread 1794618: 
2025-07-17T11:07:25.8770199Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8770405Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8770617Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8770817Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8771313Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8771522Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8771814Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8771818Z 
2025-07-17T11:07:25.8771909Z SIGABRT(6), PID: 1794252, Thread 1794620: 
2025-07-17T11:07:25.8772384Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8772597Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8772865Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8773067Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8773628Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8773841Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8774058Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8774062Z 
2025-07-17T11:07:25.8774152Z SIGABRT(6), PID: 1794252, Thread 1794622: 
2025-07-17T11:07:25.8774689Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8774901Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8775110Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8775309Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8775811Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8776078Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8776301Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8776307Z 
2025-07-17T11:07:25.8776405Z SIGABRT(6), PID: 1794252, Thread 1794624: 
2025-07-17T11:07:25.8776874Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8777090Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8777297Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8777499Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8777990Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8778205Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8778415Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8778421Z 
2025-07-17T11:07:25.8778517Z SIGABRT(6), PID: 1794252, Thread 1794626: 
2025-07-17T11:07:25.8778985Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8779198Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8779407Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8779608Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8780217Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8780445Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8780665Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8780669Z 
2025-07-17T11:07:25.8780765Z SIGABRT(6), PID: 1794252, Thread 1794629: 
2025-07-17T11:07:25.8781324Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8781555Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8781838Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8782051Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8782580Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8782879Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8783093Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8783097Z 
2025-07-17T11:07:25.8783198Z SIGABRT(6), PID: 1794252, Thread 1794632: 
2025-07-17T11:07:25.8783674Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8783892Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8784100Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8784311Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8784811Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8785030Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8785243Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8785247Z 
2025-07-17T11:07:25.8785345Z SIGABRT(6), PID: 1794252, Thread 1794635: 
2025-07-17T11:07:25.8785812Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8786027Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8786235Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8786441Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8786938Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8787152Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8787369Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8787373Z 
2025-07-17T11:07:25.8787463Z SIGABRT(6), PID: 1794252, Thread 1794637: 
2025-07-17T11:07:25.8787940Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8788145Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8788427Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8788624Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8789126Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8789415Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8789633Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8789638Z 
2025-07-17T11:07:25.8789727Z SIGABRT(6), PID: 1794252, Thread 1794640: 
2025-07-17T11:07:25.8790262Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8790482Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8790694Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8790959Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8791468Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8791678Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8791894Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8791898Z 
2025-07-17T11:07:25.8791988Z SIGABRT(6), PID: 1794252, Thread 1794643: 
2025-07-17T11:07:25.8792457Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8792666Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8792874Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8793074Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8793575Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8793782Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8794004Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8794007Z 
2025-07-17T11:07:25.8794100Z SIGABRT(6), PID: 1794252, Thread 1794646: 
2025-07-17T11:07:25.8794576Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8794786Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8795001Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8795202Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8795706Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8795913Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8796131Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8796135Z 
2025-07-17T11:07:25.8796298Z SIGABRT(6), PID: 1794252, Thread 1794649: 
2025-07-17T11:07:25.8796771Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8796985Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8797192Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8797450Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8797954Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8798170Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8798438Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8798444Z 
2025-07-17T11:07:25.8798545Z SIGABRT(6), PID: 1794252, Thread 1794652: 
2025-07-17T11:07:25.8799016Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8799297Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8799508Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8799714Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8800212Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8800428Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8800639Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8800642Z 
2025-07-17T11:07:25.8800738Z SIGABRT(6), PID: 1794252, Thread 1794655: 
2025-07-17T11:07:25.8801205Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8801421Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8801628Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8801834Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8802330Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8802553Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8802764Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8802769Z 
2025-07-17T11:07:25.8802864Z SIGABRT(6), PID: 1794252, Thread 1794658: 
2025-07-17T11:07:25.8803326Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8803540Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8803746Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8803955Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8804452Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8804739Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8804958Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8804964Z 
2025-07-17T11:07:25.8805062Z SIGABRT(6), PID: 1794252, Thread 1794661: 
2025-07-17T11:07:25.8805531Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8805803Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8806014Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8806216Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8806768Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8806990Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8807200Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8807259Z 
2025-07-17T11:07:25.8807362Z SIGABRT(6), PID: 1794252, Thread 1794664: 
2025-07-17T11:07:25.8807833Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8808050Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8808265Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8808468Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8808970Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8809180Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8809404Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8809408Z 
2025-07-17T11:07:25.8809498Z SIGABRT(6), PID: 1794249, Thread 1794249: 
2025-07-17T11:07:25.8809976Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8810183Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8810424Z frame #2: <unknown function> + 0x7741f (0x7f43fef3441f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8810658Z frame #3: <unknown function> + 0x76c8e (0x7f43fef33c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8810890Z frame #4: <unknown function> + 0x6ad69 (0x7f43fef27d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8811108Z frame #5: <unknown function> + 0x3f54db (0x7f448fb8c4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8811327Z frame #6: <unknown function> + 0x3e11d8 (0x7f448fb781d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8811539Z frame #7: <unknown function> + 0xb2f86 (0x7f448f849f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8811754Z frame #8: <unknown function> + 0x8e501 (0x7f448f825501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8812140Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f44ba62c7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.8812535Z frame #10: <unknown function> + 0xd307d9 (0x7f44d8b7f7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8812768Z frame #11: <unknown function> + 0x139902 (0x555dfb202902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8813096Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x555dfb1f9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8813330Z frame #13: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8813583Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x555dfb1f9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8813869Z frame #15: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8814127Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8814355Z frame #17: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8814656Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8814885Z frame #19: <unknown function> + 0x1474b2 (0x555dfb2104b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8815098Z frame #20: PyObject_Call + 0xbc (0x555dfb210e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8815324Z frame #21: <unknown function> + 0x1ff69d (0x555dfb2c869d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8815616Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x555dfb1fe023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8815862Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x555dfb1f9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8816174Z frame #24: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8816415Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x555dfb1f53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8816644Z frame #26: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8816882Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x555dfb1f53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8817114Z frame #28: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8817358Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x555dfb1f9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8817621Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x555dfb1fd4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8817846Z frame #31: _PyObject_Call_Prepend + 0x69 (0x555dfb20eab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8818071Z frame #32: <unknown function> + 0x205949 (0x555dfb2ce949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8818302Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x555dfb1fe023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8818551Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x555dfb1f9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8818779Z frame #35: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8819028Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x555dfb1f53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8819250Z frame #37: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8819496Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x555dfb1f53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8819720Z frame #39: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8819962Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x555dfb1f53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8820184Z frame #41: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8820436Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x555dfb1f63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8820660Z frame #43: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8821181Z frame #44: <unknown function> + 0x7ab3f7 (0x7f44d85fa3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8821418Z frame #45: <unknown function> + 0x13bc88 (0x555dfb204c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8821635Z frame #46: PyObject_Call + 0x20f (0x555dfb210f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8821958Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x555dfb1fa98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8822195Z frame #48: <unknown function> + 0x147608 (0x555dfb210608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8822444Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x555dfb1f7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8822750Z frame #50: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8822997Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x555dfb1f53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8823234Z frame #52: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8823478Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x555dfb1f63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8823786Z frame #54: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8824031Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x555dfb1f53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8824265Z frame #56: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8824507Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x555dfb1f7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8824744Z frame #58: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8824984Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x555dfb1f7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8825212Z frame #60: <unknown function> + 0x147608 (0x555dfb210608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8825452Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x555dfb1f7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8825681Z frame #62: <unknown function> + 0x147608 (0x555dfb210608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8825685Z 
2025-07-17T11:07:25.8825780Z SIGABRT(6), PID: 1794252, Thread 1794667: 
2025-07-17T11:07:25.8826276Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8826497Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8826712Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8826915Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8827430Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8827647Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8827867Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8827871Z 
2025-07-17T11:07:25.8827967Z SIGABRT(6), PID: 1794252, Thread 1794670: 
2025-07-17T11:07:25.8828452Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8828662Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8828875Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8829145Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8829651Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8829864Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8830161Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8830166Z 
2025-07-17T11:07:25.8830260Z SIGABRT(6), PID: 1794249, Thread 1794431: 
2025-07-17T11:07:25.8830742Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8831007Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8831228Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8831431Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8831935Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8832207Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8832425Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8832429Z 
2025-07-17T11:07:25.8832522Z SIGABRT(6), PID: 1794252, Thread 1794673: 
2025-07-17T11:07:25.8832998Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8833205Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8833420Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8833620Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8834126Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8834337Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8834553Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8834647Z SIGABRT(6), PID: 1794249, Thread 1794433: 
2025-07-17T11:07:25.8835120Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8835335Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8835538Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8835739Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8836239Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8836455Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8836667Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8836672Z 
2025-07-17T11:07:25.8836676Z 
2025-07-17T11:07:25.8836774Z SIGABRT(6), PID: 1794252, Thread 1794676: 
2025-07-17T11:07:25.8837245Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8837536Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8837745Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8837951Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8838503Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8838722Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8838935Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8838939Z 
2025-07-17T11:07:25.8839092Z SIGABRT(6), PID: 1794249, Thread 1794435: 
2025-07-17T11:07:25.8839568Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8839775Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8840040Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8840249Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8840751Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8840961Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8841171Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8841175Z 
2025-07-17T11:07:25.8841274Z SIGABRT(6), PID: 1794252, Thread 1794680: 
2025-07-17T11:07:25.8841740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8841961Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8842167Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8842372Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8842869Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8843084Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8843297Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8843303Z 
2025-07-17T11:07:25.8843401Z SIGABRT(6), PID: 1794249, Thread 1794437: 
2025-07-17T11:07:25.8843870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8844084Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8844291Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8844499Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8845000Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8845214Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8845494Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8845505Z 
2025-07-17T11:07:25.8845599Z SIGABRT(6), PID: 1794252, Thread 1794682: 
2025-07-17T11:07:25.8846072Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8846282Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8846553Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8846758Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8847320Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8847537Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8847761Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8847766Z 
2025-07-17T11:07:25.8847863Z SIGABRT(6), PID: 1794249, Thread 1794439: 
2025-07-17T11:07:25.8848422Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8848635Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8848847Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8849048Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8849553Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8849763Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8849978Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8849983Z 
2025-07-17T11:07:25.8850075Z SIGABRT(6), PID: 1794252, Thread 1794685: 
2025-07-17T11:07:25.8850550Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8850761Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8850977Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8851177Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8851683Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8851894Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8852111Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8852118Z 
2025-07-17T11:07:25.8852210Z SIGABRT(6), PID: 1794249, Thread 1794441: 
2025-07-17T11:07:25.8852688Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8852893Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8853106Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8853308Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8853807Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8854093Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8854309Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8854314Z 
2025-07-17T11:07:25.8854406Z SIGABRT(6), PID: 1794252, Thread 1794688: 
2025-07-17T11:07:25.8854944Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8855158Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8855432Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8855632Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8856186Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8856482Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8856695Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8856699Z 
2025-07-17T11:07:25.8856801Z SIGABRT(6), PID: 1794249, Thread 1794443: 
2025-07-17T11:07:25.8857272Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8857484Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8857690Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8857896Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8858393Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8858605Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8858815Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8858820Z 
2025-07-17T11:07:25.8858916Z SIGABRT(6), PID: 1794252, Thread 1794691: 
2025-07-17T11:07:25.8859381Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8859602Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8859806Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8860012Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8860504Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8860720Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8860933Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8860938Z 
2025-07-17T11:07:25.8861035Z SIGABRT(6), PID: 1794249, Thread 1794445: 
2025-07-17T11:07:25.8861504Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8861714Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8862000Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8862213Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8862708Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8862993Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8863208Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8863212Z 
2025-07-17T11:07:25.8863312Z SIGABRT(6), PID: 1794252, Thread 1794694: 
2025-07-17T11:07:25.8863847Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8864067Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8864277Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8864485Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8865036Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8865255Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8865467Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8865471Z 
2025-07-17T11:07:25.8865569Z SIGABRT(6), PID: 1794249, Thread 1794447: 
2025-07-17T11:07:25.8866041Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8866259Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8866465Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8866678Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8867182Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8867390Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8867605Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8867609Z 
2025-07-17T11:07:25.8867703Z SIGABRT(6), PID: 1794252, Thread 1794697: 
2025-07-17T11:07:25.8868178Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8868388Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8868605Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8868805Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8869311Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8869521Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8869742Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8869747Z 
2025-07-17T11:07:25.8869839Z SIGABRT(6), PID: 1794249, Thread 1794450: 
2025-07-17T11:07:25.8870379Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8870588Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8870807Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8871065Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8871571Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8871785Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8872061Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8872068Z 
2025-07-17T11:07:25.8872165Z SIGABRT(6), PID: 1794252, Thread 1794701: 
2025-07-17T11:07:25.8872640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8872905Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8873119Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8873319Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8873824Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8874033Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8874252Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8874258Z 
2025-07-17T11:07:25.8874348Z SIGABRT(6), PID: 1794249, Thread 1794451: 
2025-07-17T11:07:25.8874822Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8875028Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8875241Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8875441Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8875943Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8876153Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8876368Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8876371Z 
2025-07-17T11:07:25.8876463Z SIGABRT(6), PID: 1794252, Thread 1794704: 
2025-07-17T11:07:25.8876942Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8877164Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8877368Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8877567Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8878071Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8878359Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8878572Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8878576Z 
2025-07-17T11:07:25.8878677Z SIGABRT(6), PID: 1794252, Thread 1794707: 
2025-07-17T11:07:25.8879144Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8879416Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8879628Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8879833Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8880401Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8880627Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8880837Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8880900Z 
2025-07-17T11:07:25.8880999Z SIGABRT(6), PID: 1794249, Thread 1794454: 
2025-07-17T11:07:25.8881467Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8881680Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8881887Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8882090Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8882586Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8882805Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8883018Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8883022Z 
2025-07-17T11:07:25.8883122Z SIGABRT(6), PID: 1794252, Thread 1794710: 
2025-07-17T11:07:25.8883595Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8883815Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8884021Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8884228Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8884721Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8884941Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8885159Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8885163Z 
2025-07-17T11:07:25.8885254Z SIGABRT(6), PID: 1794249, Thread 1794457: 
2025-07-17T11:07:25.8885723Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8885933Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8886144Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8886514Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8887008Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8887230Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8887442Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8887505Z 
2025-07-17T11:07:25.8887603Z SIGABRT(6), PID: 1794252, Thread 1794713: 
2025-07-17T11:07:25.8888076Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8888343Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8888559Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8888764Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8889264Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8889534Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8889758Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8889762Z 
2025-07-17T11:07:25.8889855Z SIGABRT(6), PID: 1794249, Thread 1794460: 
2025-07-17T11:07:25.8890331Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8890543Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8890754Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8890955Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8891460Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8891671Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8891886Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8891890Z 
2025-07-17T11:07:25.8891984Z SIGABRT(6), PID: 1794252, Thread 1794716: 
2025-07-17T11:07:25.8892461Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8892670Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8892890Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8893091Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8893595Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8893804Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8894024Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8894028Z 
2025-07-17T11:07:25.8894121Z SIGABRT(6), PID: 1794250, Thread 1794250: 
2025-07-17T11:07:25.8894605Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8894887Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8895136Z frame #2: <unknown function> + 0x7741f (0x7fcaf024341f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8895374Z frame #3: <unknown function> + 0x76c8e (0x7fcaf0242c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8895610Z frame #4: <unknown function> + 0x6ad69 (0x7fcaf0236d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8895885Z frame #5: <unknown function> + 0x3f54db (0x7fcb80e9b4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8896202Z frame #6: <unknown function> + 0x3e11d8 (0x7fcb80e871d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8896420Z frame #7: <unknown function> + 0xb2f86 (0x7fcb80b58f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8896706Z frame #8: <unknown function> + 0x8e501 (0x7fcb80b34501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8897102Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fcbab93b7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.8897504Z frame #10: <unknown function> + 0xd307d9 (0x7fcbc9e8e7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8897807Z frame #11: <unknown function> + 0x139902 (0x55c8f763a902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8898071Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55c8f7631a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8898304Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8898555Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55c8f7631a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8898784Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8899034Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8899263Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8899508Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8899734Z frame #19: <unknown function> + 0x1474b2 (0x55c8f76484b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8899949Z frame #20: PyObject_Call + 0xbc (0x55c8f7648e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8900173Z frame #21: <unknown function> + 0x1ff69d (0x55c8f770069d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8900408Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55c8f7636023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8900650Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55c8f7631cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8900880Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8901121Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55c8f762d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8901358Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8901601Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55c8f762d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8901831Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8902075Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55c8f7631a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8902338Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55c8f76354b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8902564Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55c8f7646ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8902792Z frame #32: <unknown function> + 0x205949 (0x55c8f7706949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8903099Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55c8f7636023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8903351Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55c8f7631cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8903578Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8903819Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55c8f762d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8904117Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8904367Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55c8f762d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8904591Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8904892Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55c8f762d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8905121Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8905369Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55c8f762e3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8905654Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8906050Z frame #44: <unknown function> + 0x7ab3f7 (0x7fcbc99093f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8906278Z frame #45: <unknown function> + 0x13bc88 (0x55c8f763cc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8906492Z frame #46: PyObject_Call + 0x20f (0x55c8f7648f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8906740Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55c8f763298d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8906968Z frame #48: <unknown function> + 0x147608 (0x55c8f7648608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8907214Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c8f762fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8907448Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8907689Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55c8f762d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8907921Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8908162Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55c8f762e3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8908403Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8908646Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55c8f762d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8908880Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8909124Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c8f762fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8909362Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8909607Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c8f762fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8909838Z frame #60: <unknown function> + 0x147608 (0x55c8f7648608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8910083Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c8f762fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8910308Z frame #62: <unknown function> + 0x147608 (0x55c8f7648608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8910312Z 
2025-07-17T11:07:25.8910407Z SIGABRT(6), PID: 1794249, Thread 1794463: 
2025-07-17T11:07:25.8910897Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8911198Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8911415Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8911626Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8912223Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8912442Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8912665Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8912669Z 
2025-07-17T11:07:25.8912821Z SIGABRT(6), PID: 1794252, Thread 1794718: 
2025-07-17T11:07:25.8913301Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8913520Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8913800Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8914002Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8914514Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8914726Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8914957Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8914961Z 
2025-07-17T11:07:25.8915058Z SIGABRT(6), PID: 1794250, Thread 1794448: 
2025-07-17T11:07:25.8915537Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8915757Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8915964Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8916175Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8916675Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8916890Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8917107Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8917113Z 
2025-07-17T11:07:25.8917213Z SIGABRT(6), PID: 1794249, Thread 1794466: 
2025-07-17T11:07:25.8917682Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8917900Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8918104Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8918312Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8918810Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8919029Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8919242Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8919311Z 
2025-07-17T11:07:25.8919416Z SIGABRT(6), PID: 1794252, Thread 1794720: 
2025-07-17T11:07:25.8919887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8920107Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8920373Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8920588Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8921139Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8921357Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8921572Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8921576Z 
2025-07-17T11:07:25.8921680Z SIGABRT(6), PID: 1794250, Thread 1794453: 
2025-07-17T11:07:25.8922209Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8922430Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8922638Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8922846Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8923342Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8923558Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8923766Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8923868Z SIGABRT(6), PID: 1794252, Thread 1794722: 
2025-07-17T11:07:25.8924339Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8924560Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8924766Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8924974Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8925474Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8925687Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8925900Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8926000Z SIGABRT(6), PID: 1794249, Thread 1794470: 
2025-07-17T11:07:25.8926473Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8926685Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8926890Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8927099Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8927608Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8927893Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8928107Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8928113Z 
2025-07-17T11:07:25.8928117Z 
2025-07-17T11:07:25.8928120Z 
2025-07-17T11:07:25.8928214Z SIGABRT(6), PID: 1794250, Thread 1794456: 
2025-07-17T11:07:25.8928774Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8928987Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8929203Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8929461Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8929967Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8930179Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8930458Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8930463Z 
2025-07-17T11:07:25.8930557Z SIGABRT(6), PID: 1794252, Thread 1794724: 
2025-07-17T11:07:25.8931037Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8931248Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8931465Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8931666Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8932173Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8932383Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8932602Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8932697Z SIGABRT(6), PID: 1794249, Thread 1794473: 
2025-07-17T11:07:25.8933172Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8933385Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8933601Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8933800Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8934309Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8934519Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8934739Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8934743Z 
2025-07-17T11:07:25.8934746Z 
2025-07-17T11:07:25.8934838Z SIGABRT(6), PID: 1794252, Thread 1794726: 
2025-07-17T11:07:25.8935316Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8935526Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8935816Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8936067Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8936575Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8936865Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8937079Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8937084Z 
2025-07-17T11:07:25.8937182Z SIGABRT(6), PID: 1794250, Thread 1794459: 
2025-07-17T11:07:25.8937723Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8937948Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8938159Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8938367Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8938950Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8939170Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8939383Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8939387Z 
2025-07-17T11:07:25.8939488Z SIGABRT(6), PID: 1794249, Thread 1794476: 
2025-07-17T11:07:25.8939963Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8940184Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8940388Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8940599Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8941099Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8941317Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8941530Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8941534Z 
2025-07-17T11:07:25.8941641Z SIGABRT(6), PID: 1794250, Thread 1794462: 
2025-07-17T11:07:25.8942112Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8942328Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8942535Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8942743Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8943239Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8943456Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8943665Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8943669Z 
2025-07-17T11:07:25.8943770Z SIGABRT(6), PID: 1794252, Thread 1794728: 
2025-07-17T11:07:25.8944354Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8944576Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8944784Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8944993Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8945558Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8945790Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8946064Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8946071Z 
2025-07-17T11:07:25.8946176Z SIGABRT(6), PID: 1794249, Thread 1794479: 
2025-07-17T11:07:25.8946649Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8946927Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8947147Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8947349Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8947852Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8948066Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8948279Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8948284Z 
2025-07-17T11:07:25.8948379Z SIGABRT(6), PID: 1794252, Thread 1794730: 
2025-07-17T11:07:25.8948858Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8949068Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8949281Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8949481Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8949984Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8950199Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8950419Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8950423Z 
2025-07-17T11:07:25.8950513Z SIGABRT(6), PID: 1794250, Thread 1794465: 
2025-07-17T11:07:25.8950999Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8951208Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8951422Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8951625Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8952131Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8952415Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8952631Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8952635Z 
2025-07-17T11:07:25.8952731Z SIGABRT(6), PID: 1794249, Thread 1794482: 
2025-07-17T11:07:25.8953213Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8953478Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8953694Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8953891Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8954457Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8954669Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8954887Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8955039Z SIGABRT(6), PID: 1794250, Thread 1794468: 
2025-07-17T11:07:25.8955526Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8955739Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8955957Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8956162Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8956673Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8956890Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8957110Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8957205Z SIGABRT(6), PID: 1794252, Thread 1794732: 
2025-07-17T11:07:25.8957682Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8957895Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8958107Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8958313Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8958815Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8959025Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8959244Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8959251Z 
2025-07-17T11:07:25.8959254Z 
2025-07-17T11:07:25.8959257Z 
2025-07-17T11:07:25.8959355Z SIGABRT(6), PID: 1794247, Thread 1794247: 
2025-07-17T11:07:25.8959825Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8960042Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8960282Z frame #2: <unknown function> + 0x7744b (0x7f54ea0ca44b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8960522Z frame #3: <unknown function> + 0x76c8e (0x7f54ea0c9c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8960833Z frame #4: <unknown function> + 0x6ad69 (0x7f54ea0bdd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8961062Z frame #5: <unknown function> + 0x3f54db (0x7f557ad224db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8961280Z frame #6: <unknown function> + 0x3e11d8 (0x7f557ad0e1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8961562Z frame #7: <unknown function> + 0xb2f86 (0x7f557a9dff86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8961774Z frame #8: <unknown function> + 0x8e501 (0x7f557a9bb501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8962168Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f55a57c27ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.8962614Z frame #10: <unknown function> + 0xd307d9 (0x7f55c3d117d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8962854Z frame #11: <unknown function> + 0x139902 (0x56305eaa2902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8963111Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x56305ea99a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8963412Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8963657Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x56305ea99a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8963891Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8964135Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8964368Z frame #17: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8964610Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8964843Z frame #19: <unknown function> + 0x1474b2 (0x56305eab04b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8965051Z frame #20: PyObject_Call + 0xbc (0x56305eab0e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8965286Z frame #21: <unknown function> + 0x1ff69d (0x56305eb6869d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8965511Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x56305ea9e023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8965763Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x56305ea99cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8965991Z frame #24: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8966237Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x56305ea953bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8966464Z frame #26: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8966710Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x56305ea953bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8966937Z frame #28: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8967189Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x56305ea99a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8967450Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x56305ea9d4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8967681Z frame #31: _PyObject_Call_Prepend + 0x69 (0x56305eaaeab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8967905Z frame #32: <unknown function> + 0x205949 (0x56305eb6e949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8968138Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x56305ea9e023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8968385Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x56305ea99cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8968629Z frame #35: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8968939Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x56305ea953bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8969172Z frame #37: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8969411Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x56305ea953bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8969698Z frame #39: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8969940Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x56305ea953bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8970174Z frame #41: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8970470Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x56305ea963f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8970706Z frame #43: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8971098Z frame #44: <unknown function> + 0x7ab3f7 (0x7f55c378c3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8971410Z frame #45: <unknown function> + 0x13bc88 (0x56305eaa4c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8971618Z frame #46: PyObject_Call + 0x20f (0x56305eab0f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8971871Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x56305ea9a98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8972094Z frame #48: <unknown function> + 0x147608 (0x56305eab0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8972341Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x56305ea97c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8972570Z frame #50: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8972817Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x56305ea953bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8973047Z frame #52: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8973297Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x56305ea963f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8973522Z frame #54: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8973764Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x56305ea953bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8973989Z frame #56: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8974235Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x56305ea97c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8974464Z frame #58: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8974716Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x56305ea97c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8974942Z frame #60: <unknown function> + 0x147608 (0x56305eab0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8975191Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x56305ea97c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8975415Z frame #62: <unknown function> + 0x147608 (0x56305eab0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8975418Z 
2025-07-17T11:07:25.8975517Z SIGABRT(6), PID: 1794252, Thread 1794735: 
2025-07-17T11:07:25.8976049Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8976275Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8976489Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8976787Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8977296Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8977518Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8977730Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8977805Z 
2025-07-17T11:07:25.8977912Z SIGABRT(6), PID: 1794250, Thread 1794469: 
2025-07-17T11:07:25.8978385Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8978674Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8978889Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8979105Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8979609Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8979902Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8980118Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8980122Z 
2025-07-17T11:07:25.8980219Z SIGABRT(6), PID: 1794249, Thread 1794484: 
2025-07-17T11:07:25.8980687Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8980910Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8981117Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8981327Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8981826Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8988653Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8988932Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8988938Z 
2025-07-17T11:07:25.8989051Z SIGABRT(6), PID: 1794247, Thread 1794287: 
2025-07-17T11:07:25.8989558Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8989787Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8989996Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8990203Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8990715Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.8990932Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8991146Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8991246Z SIGABRT(6), PID: 1794246, Thread 1794246: 
2025-07-17T11:07:25.8991725Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.8992071Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.8992309Z frame #2: <unknown function> + 0x77435 (0x7fce17e61435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8992553Z frame #3: <unknown function> + 0x76c8e (0x7fce17e60c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8992779Z frame #4: <unknown function> + 0x6ad69 (0x7fce17e54d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.8993067Z frame #5: <unknown function> + 0x3f54db (0x7fcea8ab94db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8993283Z frame #6: <unknown function> + 0x3e11d8 (0x7fcea8aa51d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8993502Z frame #7: <unknown function> + 0xb2f86 (0x7fcea8776f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8993776Z frame #8: <unknown function> + 0x8e501 (0x7fcea8752501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.8994169Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fced35597ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.8994571Z frame #10: <unknown function> + 0xd307d9 (0x7fcef1aa87d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.8994905Z frame #11: <unknown function> + 0x139902 (0x55a9caf4b902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8995172Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55a9caf42a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8995405Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8995659Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55a9caf42a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8995887Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8996136Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8996360Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8996597Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8996827Z frame #19: <unknown function> + 0x1474b2 (0x55a9caf594b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8997030Z frame #20: PyObject_Call + 0xbc (0x55a9caf59e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8997261Z frame #21: <unknown function> + 0x1ff69d (0x55a9cb01169d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8997487Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55a9caf47023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8997734Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55a9caf42cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8997962Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8998204Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55a9caf3e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8998423Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8998664Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55a9caf3e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8998888Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8999138Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55a9caf42a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8999394Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55a9caf464b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8999622Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55a9caf57ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.8999846Z frame #32: <unknown function> + 0x205949 (0x55a9cb017949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9000147Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55a9caf47023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9000388Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55a9caf42cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9000618Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9000854Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55a9caf3e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9001142Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9001381Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55a9caf3e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9001608Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9001900Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55a9caf3e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9002132Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9002372Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55a9caf3f3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9002663Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9003058Z frame #44: <unknown function> + 0x7ab3f7 (0x7fcef15233f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9003289Z frame #45: <unknown function> + 0x13bc88 (0x55a9caf4dc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9003497Z frame #46: PyObject_Call + 0x20f (0x55a9caf59f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9003748Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55a9caf4398d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9003967Z frame #48: <unknown function> + 0x147608 (0x55a9caf59608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9004212Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a9caf40c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9004433Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9004689Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55a9caf3e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9004931Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9005188Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55a9caf3f3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9005419Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9005661Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55a9caf3e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9005885Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9006139Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a9caf40c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9006362Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9006612Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a9caf40c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9006837Z frame #60: <unknown function> + 0x147608 (0x55a9caf59608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9007089Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a9caf40c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9007311Z frame #62: <unknown function> + 0x147608 (0x55a9caf59608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9007321Z 
2025-07-17T11:07:25.9007326Z 
2025-07-17T11:07:25.9007426Z SIGABRT(6), PID: 1794252, Thread 1794737: 
2025-07-17T11:07:25.9007921Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9008213Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9008436Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9008644Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9009221Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9009438Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9009662Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9009667Z 
2025-07-17T11:07:25.9009842Z SIGABRT(6), PID: 1794250, Thread 1794472: 
2025-07-17T11:07:25.9010323Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9010534Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9010805Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9011010Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9011522Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9011732Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9011949Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9011954Z 
2025-07-17T11:07:25.9012046Z SIGABRT(6), PID: 1794249, Thread 1794486: 
2025-07-17T11:07:25.9012524Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9012732Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9012941Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9013145Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9013645Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9013855Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9014072Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9014078Z 
2025-07-17T11:07:25.9014168Z SIGABRT(6), PID: 1794252, Thread 1794739: 
2025-07-17T11:07:25.9014645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9014855Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9015067Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9015268Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9015761Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9016056Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9016360Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9016365Z 
2025-07-17T11:07:25.9016457Z SIGABRT(6), PID: 1794247, Thread 1794289: 
2025-07-17T11:07:25.9016926Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9017134Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9017425Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9017635Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9018207Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9018425Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9018636Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9018726Z SIGABRT(6), PID: 1794246, Thread 1794260: 
2025-07-17T11:07:25.9019369Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9019585Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9019793Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9019996Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9020491Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9020706Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9020914Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9020919Z 
2025-07-17T11:07:25.9020924Z 
2025-07-17T11:07:25.9021016Z SIGABRT(6), PID: 1794250, Thread 1794475: 
2025-07-17T11:07:25.9021477Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9021693Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9021903Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9022112Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9022604Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9022822Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9023034Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9023039Z 
2025-07-17T11:07:25.9023133Z SIGABRT(6), PID: 1794249, Thread 1794488: 
2025-07-17T11:07:25.9023599Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9023813Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9024017Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9024219Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9024711Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9024996Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9025210Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9025214Z 
2025-07-17T11:07:25.9025311Z SIGABRT(6), PID: 1794252, Thread 1794741: 
2025-07-17T11:07:25.9025837Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9026055Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9026316Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9026523Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9027019Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9027289Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9027501Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9027506Z 
2025-07-17T11:07:25.9027601Z SIGABRT(6), PID: 1794246, Thread 1794261: 
2025-07-17T11:07:25.9028069Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9028281Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9028491Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9028706Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9029211Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9029423Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9029639Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9029730Z SIGABRT(6), PID: 1794247, Thread 1794292: 
2025-07-17T11:07:25.9030203Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9030414Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9030621Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9030819Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9031324Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9031535Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9031754Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9031845Z SIGABRT(6), PID: 1794250, Thread 1794478: 
2025-07-17T11:07:25.9032320Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9032531Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9032818Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9033016Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9033516Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9033725Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9033998Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9034004Z 
2025-07-17T11:07:25.9034007Z 
2025-07-17T11:07:25.9034010Z 
2025-07-17T11:07:25.9034101Z SIGABRT(6), PID: 1794252, Thread 1794743: 
2025-07-17T11:07:25.9034629Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9034845Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9035061Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9035260Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9035821Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9036035Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9036249Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9036253Z 
2025-07-17T11:07:25.9036344Z SIGABRT(6), PID: 1794249, Thread 1794490: 
2025-07-17T11:07:25.9036822Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9037032Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9037242Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9037445Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9037945Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9038152Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9038369Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9038461Z SIGABRT(6), PID: 1794250, Thread 1794481: 
2025-07-17T11:07:25.9038935Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9039146Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9039356Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9039556Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9040060Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9040268Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9040479Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9040572Z SIGABRT(6), PID: 1794247, Thread 1794294: 
2025-07-17T11:07:25.9041039Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9041329Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9041535Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9041738Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9042311Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9042524Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9042796Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9042900Z SIGABRT(6), PID: 1794252, Thread 1794745: 
2025-07-17T11:07:25.9043372Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9043592Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9043858Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9044058Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9044552Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9044763Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9044974Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9045071Z SIGABRT(6), PID: 1794246, Thread 1794262: 
2025-07-17T11:07:25.9045539Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9045754Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9045957Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9046163Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9046660Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9046880Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9047091Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9047096Z 
2025-07-17T11:07:25.9047100Z 
2025-07-17T11:07:25.9047105Z 
2025-07-17T11:07:25.9047108Z 
2025-07-17T11:07:25.9047111Z 
2025-07-17T11:07:25.9047207Z SIGABRT(6), PID: 1794250, Thread 1794483: 
2025-07-17T11:07:25.9047670Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9047886Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9048095Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9048303Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9048806Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9049087Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9049297Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9049302Z 
2025-07-17T11:07:25.9049397Z SIGABRT(6), PID: 1794252, Thread 1794747: 
2025-07-17T11:07:25.9049864Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9050131Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9050345Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9050547Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9051105Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9051318Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9051534Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9051679Z SIGABRT(6), PID: 1794249, Thread 1794492: 
2025-07-17T11:07:25.9052152Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9052357Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9052567Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9052765Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9053266Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9053477Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9053690Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9053695Z 
2025-07-17T11:07:25.9053698Z 
2025-07-17T11:07:25.9053788Z SIGABRT(6), PID: 1794246, Thread 1794263: 
2025-07-17T11:07:25.9054264Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9054473Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9054684Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9054889Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9055388Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9055596Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9055818Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9055954Z SIGABRT(6), PID: 1794247, Thread 1794296: 
2025-07-17T11:07:25.9056422Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9056628Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9056840Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9057039Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9057626Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9057836Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9058051Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9058055Z 
2025-07-17T11:07:25.9058125Z 
2025-07-17T11:07:25.9058218Z SIGABRT(6), PID: 1794250, Thread 1794485: 
2025-07-17T11:07:25.9058686Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9058963Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9059178Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9059380Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9059887Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9060170Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9060390Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9060396Z 
2025-07-17T11:07:25.9060484Z SIGABRT(6), PID: 1794252, Thread 1794749: 
2025-07-17T11:07:25.9060957Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9061171Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9061379Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9061580Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9062074Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9062289Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9062497Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9062501Z 
2025-07-17T11:07:25.9062595Z SIGABRT(6), PID: 1794249, Thread 1794494: 
2025-07-17T11:07:25.9063064Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9063272Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9063476Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9063683Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9064178Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9064391Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9064607Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9064611Z 
2025-07-17T11:07:25.9064702Z SIGABRT(6), PID: 1794246, Thread 1794264: 
2025-07-17T11:07:25.9065171Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9065461Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9065668Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9065872Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9066427Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9066644Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9066854Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9066858Z 
2025-07-17T11:07:25.9067008Z SIGABRT(6), PID: 1794247, Thread 1794298: 
2025-07-17T11:07:25.9067478Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9067694Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9067895Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9068156Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9068650Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9068865Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9069075Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9069082Z 
2025-07-17T11:07:25.9069175Z SIGABRT(6), PID: 1794250, Thread 1794487: 
2025-07-17T11:07:25.9069642Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9069849Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9070062Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9070266Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9070762Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9070971Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9071182Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9071194Z 
2025-07-17T11:07:25.9071283Z SIGABRT(6), PID: 1794252, Thread 1794751: 
2025-07-17T11:07:25.9071751Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9071959Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9072170Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9072367Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9072864Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9073072Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9073285Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9073377Z 
2025-07-17T11:07:25.9073474Z SIGABRT(6), PID: 1794246, Thread 1794265: 
2025-07-17T11:07:25.9073943Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9074152Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9074428Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9074629Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9075124Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9075409Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9075639Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9075729Z SIGABRT(6), PID: 1794249, Thread 1794496: 
2025-07-17T11:07:25.9076200Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9076473Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9076685Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9076885Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9077387Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9077593Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9077811Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9077815Z 
2025-07-17T11:07:25.9077819Z 
2025-07-17T11:07:25.9077909Z SIGABRT(6), PID: 1794247, Thread 1794300: 
2025-07-17T11:07:25.9078384Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9078589Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9078800Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9078995Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9079498Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9079712Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9079931Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9079937Z 
2025-07-17T11:07:25.9080030Z SIGABRT(6), PID: 1794252, Thread 1794753: 
2025-07-17T11:07:25.9080509Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9080717Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9080931Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9081132Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9081634Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9081909Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9082125Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9082132Z 
2025-07-17T11:07:25.9082224Z SIGABRT(6), PID: 1794250, Thread 1794489: 
2025-07-17T11:07:25.9082750Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9082969Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9083181Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9083443Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9083942Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9084159Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9084430Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9084435Z 
2025-07-17T11:07:25.9084530Z SIGABRT(6), PID: 1794247, Thread 1794302: 
2025-07-17T11:07:25.9084999Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9085213Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9085421Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9085628Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9086130Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9086347Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9086560Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9086564Z 
2025-07-17T11:07:25.9086666Z SIGABRT(6), PID: 1794249, Thread 1794498: 
2025-07-17T11:07:25.9087132Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9087346Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9087552Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9087754Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9088248Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9088470Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9088681Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9088783Z SIGABRT(6), PID: 1794246, Thread 1794266: 
2025-07-17T11:07:25.9089249Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9089470Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9089676Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9089952Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9090446Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9090659Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9090930Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9090934Z 
2025-07-17T11:07:25.9091032Z SIGABRT(6), PID: 1794252, Thread 1794755: 
2025-07-17T11:07:25.9091554Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9091771Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9091983Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9092189Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9092747Z frame #4: <unknown function> + 0x3475ab (0x7fe189ae95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9092962Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9093176Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9093185Z 
2025-07-17T11:07:25.9093189Z 
2025-07-17T11:07:25.9093282Z SIGABRT(6), PID: 1794250, Thread 1794491: 
2025-07-17T11:07:25.9093756Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9093969Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9094182Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9094384Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9094890Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9095102Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9095323Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9095327Z 
2025-07-17T11:07:25.9095418Z SIGABRT(6), PID: 1794252, Thread 1795306: 
2025-07-17T11:07:25.9095891Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9096402Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fe09b34908a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9096620Z frame #2: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9096815Z frame #3: pthread_kill + 0x12c (0x7fe18cc539fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9096989Z frame #4: raise + 0x16 (0x7fe18cbff476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9097152Z frame #5: abort + 0xd3 (0x7fe18cbe57f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9097400Z frame #6: <unknown function> + 0x972cc (0x7fe008ed22cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9097633Z frame #7: <unknown function> + 0x95f47 (0x7fe008ed0f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9097862Z frame #8: <unknown function> + 0x34b71 (0x7fe008e6fb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9098159Z frame #9: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9098383Z frame #10: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9098391Z 
2025-07-17T11:07:25.9098486Z SIGABRT(6), PID: 1794247, Thread 1794304: 
2025-07-17T11:07:25.9099036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9099254Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9099467Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9099740Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9100245Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9100456Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9100768Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9100772Z 
2025-07-17T11:07:25.9100866Z SIGABRT(6), PID: 1794249, Thread 1794500: 
2025-07-17T11:07:25.9101362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9101572Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9101781Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9101985Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9102496Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9102706Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9102922Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9102926Z 
2025-07-17T11:07:25.9103020Z SIGABRT(6), PID: 1794246, Thread 1794267: 
2025-07-17T11:07:25.9103498Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9103709Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9103927Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9104130Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9104634Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9104853Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9105076Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9105084Z 
2025-07-17T11:07:25.9105176Z SIGABRT(6), PID: 1794250, Thread 1794493: 
2025-07-17T11:07:25.9105649Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9105867Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9106072Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9106355Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9106855Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9107072Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9107341Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9107346Z 
2025-07-17T11:07:25.9107445Z SIGABRT(6), PID: 1794252, Thread 1798434: 
2025-07-17T11:07:25.9107914Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9108191Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9108372Z frame #2: __poll + 0x4f (0x7fe18ccd5bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9108788Z frame #3: <unknown function> + 0x489f (0x7fe18c7e189f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.9109078Z frame #4: <unknown function> + 0x1483bc (0x558910bc03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9109338Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9109572Z frame #6: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9109813Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9110039Z frame #8: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9110294Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x558910ba8a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9110528Z frame #10: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9110782Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x558910ba6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9111010Z frame #12: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9111253Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9111480Z frame #14: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9111724Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9111943Z frame #16: <unknown function> + 0x147734 (0x558910bbf734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9112170Z frame #17: <unknown function> + 0x21ca82 (0x558910c94a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9112392Z frame #18: <unknown function> + 0x21c9e4 (0x558910c949e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9112614Z frame #19: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9112832Z frame #20: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9112838Z 
2025-07-17T11:07:25.9112941Z SIGABRT(6), PID: 1794247, Thread 1794307: 
2025-07-17T11:07:25.9113423Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9113646Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9113852Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9114065Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9114636Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9114853Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9115067Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9115072Z 
2025-07-17T11:07:25.9115171Z SIGABRT(6), PID: 1794249, Thread 1794502: 
2025-07-17T11:07:25.9115700Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9115919Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9116179Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9116389Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9116889Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9117162Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9117374Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9117381Z 
2025-07-17T11:07:25.9117479Z SIGABRT(6), PID: 1794250, Thread 1794495: 
2025-07-17T11:07:25.9117950Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9118167Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9118374Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9118583Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9119083Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9119303Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9119515Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9119519Z 
2025-07-17T11:07:25.9119616Z SIGABRT(6), PID: 1794246, Thread 1794268: 
2025-07-17T11:07:25.9120092Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9120305Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9120518Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9120721Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9121228Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9121440Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9121660Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9121664Z 
2025-07-17T11:07:25.9121758Z SIGABRT(6), PID: 1794247, Thread 1794309: 
2025-07-17T11:07:25.9122233Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9122442Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9122723Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9122924Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9123433Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9123702Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9123919Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9123924Z 
2025-07-17T11:07:25.9124015Z SIGABRT(6), PID: 1794249, Thread 1794504: 
2025-07-17T11:07:25.9124551Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9124764Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9124976Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9125236Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9125743Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9125953Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9126172Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9126176Z 
2025-07-17T11:07:25.9126269Z SIGABRT(6), PID: 1794250, Thread 1794497: 
2025-07-17T11:07:25.9126745Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9126955Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9127172Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9127374Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9127879Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9128089Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9128308Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9128313Z 
2025-07-17T11:07:25.9128405Z SIGABRT(6), PID: 1794246, Thread 1794269: 
2025-07-17T11:07:25.9128884Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9129095Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9129314Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9129515Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9130023Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9130236Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9130459Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9130529Z 
2025-07-17T11:07:25.9130626Z SIGABRT(6), PID: 1794250, Thread 1794499: 
2025-07-17T11:07:25.9131107Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9131323Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9131531Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9131800Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9132305Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9132598Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9132818Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9132825Z 
2025-07-17T11:07:25.9132925Z SIGABRT(6), PID: 1794249, Thread 1794506: 
2025-07-17T11:07:25.9133393Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9133665Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9133871Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9134079Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9134573Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9134795Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9135008Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9135108Z SIGABRT(6), PID: 1794247, Thread 1794311: 
2025-07-17T11:07:25.9135575Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9135793Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9136062Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9136269Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9136770Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9136985Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9137198Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9137202Z 
2025-07-17T11:07:25.9137206Z 
2025-07-17T11:07:25.9137303Z SIGABRT(6), PID: 1794246, Thread 1794270: 
2025-07-17T11:07:25.9137769Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9137987Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9138195Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9138409Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9138907Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9139209Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9139423Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9139429Z 
2025-07-17T11:07:25.9139525Z SIGABRT(6), PID: 1794252, Thread 1798435: 
2025-07-17T11:07:25.9140062Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9140279Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9140487Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9140791Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fe18cc512dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9141241Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fe0c7091048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9141676Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fe0c7093bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9141985Z frame #6: <unknown function> + 0xdc253 (0x7fe0995b5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9142202Z frame #7: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9142415Z frame #8: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9142426Z 
2025-07-17T11:07:25.9142518Z SIGABRT(6), PID: 1794251, Thread 1794251: 
2025-07-17T11:07:25.9143001Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9143208Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9143449Z frame #2: <unknown function> + 0x773e7 (0x7f5ec10613e7 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9143682Z frame #3: <unknown function> + 0x76c8e (0x7f5ec1060c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9143919Z frame #4: <unknown function> + 0x6ad69 (0x7f5ec1054d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9144136Z frame #5: <unknown function> + 0x3f54db (0x7f5f51cb94db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.9144359Z frame #6: <unknown function> + 0x3e11d8 (0x7f5f51ca51d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.9144574Z frame #7: <unknown function> + 0xb2f86 (0x7f5f51976f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.9144788Z frame #8: <unknown function> + 0x8e501 (0x7f5f51952501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:25.9145175Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f5f7c7597ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:25.9145568Z frame #10: <unknown function> + 0xd307d9 (0x7f5f9aca87d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9145797Z frame #11: <unknown function> + 0x139902 (0x55d53e598902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9146057Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55d53e58fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9146288Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9146536Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55d53e58fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9146763Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9147019Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9147247Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9147563Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9147787Z frame #19: <unknown function> + 0x1474b2 (0x55d53e5a64b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9148004Z frame #20: PyObject_Call + 0xbc (0x55d53e5a6e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9148288Z frame #21: <unknown function> + 0x1ff69d (0x55d53e65e69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9148523Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55d53e594023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9148766Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55d53e58fcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9149150Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9149393Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55d53e58b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9149631Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9149868Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55d53e58b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9150157Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9150404Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55d53e58fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9150670Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55d53e5934b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9150895Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55d53e5a4ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9151124Z frame #32: <unknown function> + 0x205949 (0x55d53e664949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9151348Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55d53e594023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9151604Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55d53e58fcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9151834Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9152081Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55d53e58b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9152306Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9152548Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55d53e58b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9152774Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9153022Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55d53e58b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9153245Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9153497Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55d53e58c3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9153723Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9154114Z frame #44: <unknown function> + 0x7ab3f7 (0x7f5f9a7233f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9154345Z frame #45: <unknown function> + 0x13bc88 (0x55d53e59ac88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9154558Z frame #46: PyObject_Call + 0x20f (0x55d53e5a6f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9154806Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55d53e59098d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9155036Z frame #48: <unknown function> + 0x147608 (0x55d53e5a6608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9155278Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d53e58dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9155589Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9155831Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55d53e58b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9156070Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9156387Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55d53e58c3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9156628Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9156864Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55d53e58b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9157151Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9157398Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d53e58dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9157634Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9157873Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d53e58dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9158159Z frame #60: <unknown function> + 0x147608 (0x55d53e5a6608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9158400Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d53e58dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9158628Z frame #62: <unknown function> + 0x147608 (0x55d53e5a6608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9158632Z 
2025-07-17T11:07:25.9158729Z SIGABRT(6), PID: 1794250, Thread 1794501: 
2025-07-17T11:07:25.9159214Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9159427Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9159646Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9159855Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9160367Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9160582Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9160801Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9160805Z 
2025-07-17T11:07:25.9160899Z SIGABRT(6), PID: 1794249, Thread 1794508: 
2025-07-17T11:07:25.9161379Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9161590Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9161801Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9162005Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9162517Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9162730Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9162947Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9162951Z 
2025-07-17T11:07:25.9163044Z SIGABRT(6), PID: 1794247, Thread 1794313: 
2025-07-17T11:07:25.9163527Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9163811Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9164027Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9164229Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9164813Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9165027Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9165250Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9165310Z 
2025-07-17T11:07:25.9165407Z SIGABRT(6), PID: 1794246, Thread 1794271: 
2025-07-17T11:07:25.9165887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9166100Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9166377Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9166581Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9167091Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9167299Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9167519Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9167525Z 
2025-07-17T11:07:25.9167622Z SIGABRT(6), PID: 1794251, Thread 1794530: 
2025-07-17T11:07:25.9168099Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9168318Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9168525Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9168729Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9169235Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9169451Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9169661Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9169668Z 
2025-07-17T11:07:25.9169765Z SIGABRT(6), PID: 1794250, Thread 1794503: 
2025-07-17T11:07:25.9170234Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9170452Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9170661Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9170870Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9171368Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9171597Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9171880Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9171884Z 
2025-07-17T11:07:25.9171985Z SIGABRT(6), PID: 1794252, Thread 1798436: 
2025-07-17T11:07:25.9172460Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9172676Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9172944Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9173183Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fe18cc512dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9173723Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fe0c708b741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9173968Z frame #5: <unknown function> + 0xdc253 (0x7fe0995b5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9174181Z frame #6: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9174399Z frame #7: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9174461Z 
2025-07-17T11:07:25.9174554Z SIGABRT(6), PID: 1794249, Thread 1794510: 
2025-07-17T11:07:25.9175037Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9175244Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9175458Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9175663Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9176228Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9176438Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9176658Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9176751Z SIGABRT(6), PID: 1794247, Thread 1794315: 
2025-07-17T11:07:25.9177226Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9177437Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9177649Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9177853Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9178357Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9178566Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9178783Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9178787Z 
2025-07-17T11:07:25.9178791Z 
2025-07-17T11:07:25.9178888Z SIGABRT(6), PID: 1794246, Thread 1794272: 
2025-07-17T11:07:25.9179370Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9179584Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9179798Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9180092Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9180595Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9180811Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9181025Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9181097Z 
2025-07-17T11:07:25.9181200Z SIGABRT(6), PID: 1794252, Thread 1798439: 
2025-07-17T11:07:25.9181674Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9181958Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9182136Z frame #2: ioctl + 0x3f (0x7fe18ccd794f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9182388Z frame #3: <unknown function> + 0x13eef0 (0x7fe008f79ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9182623Z frame #4: <unknown function> + 0x1378c4 (0x7fe008f728c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9182933Z frame #5: <unknown function> + 0x95b2b (0x7fe008ed0b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9183164Z frame #6: <unknown function> + 0x34b71 (0x7fe008e6fb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9183386Z frame #7: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9183601Z frame #8: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9183697Z SIGABRT(6), PID: 1794250, Thread 1794505: 
2025-07-17T11:07:25.9184167Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9184387Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9184595Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9184808Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9185311Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9185528Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9185739Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9185743Z 
2025-07-17T11:07:25.9185844Z SIGABRT(6), PID: 1794251, Thread 1794534: 
2025-07-17T11:07:25.9186312Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9186526Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9186735Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9186942Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9187438Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9187654Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9187865Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9187870Z 
2025-07-17T11:07:25.9187875Z 
2025-07-17T11:07:25.9187978Z SIGABRT(6), PID: 1794246, Thread 1794273: 
2025-07-17T11:07:25.9188521Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9188741Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9188949Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9189215Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9189721Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9189936Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9190202Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9190308Z SIGABRT(6), PID: 1794249, Thread 1794512: 
2025-07-17T11:07:25.9190777Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9191049Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9191255Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9191461Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9191959Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9192178Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9192388Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9192398Z 
2025-07-17T11:07:25.9192401Z 
2025-07-17T11:07:25.9192491Z SIGABRT(6), PID: 1794247, Thread 1794317: 
2025-07-17T11:07:25.9192964Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9193174Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9193389Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9193586Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9194088Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9194298Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9194518Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9194522Z 
2025-07-17T11:07:25.9194612Z SIGABRT(6), PID: 1794250, Thread 1794507: 
2025-07-17T11:07:25.9195091Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9195302Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9195517Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9195719Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9196220Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9196523Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9196745Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9196749Z 
2025-07-17T11:07:25.9196842Z SIGABRT(6), PID: 1794252, Thread 1799062: 
2025-07-17T11:07:25.9197314Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9197582Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9197762Z frame #2: __poll + 0x4f (0x7fe18ccd5bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9197979Z frame #3: <unknown function> + 0xd4540 (0x7fe0510e9540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9198255Z frame #4: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9198472Z frame #5: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9198569Z SIGABRT(6), PID: 1794251, Thread 1794537: 
2025-07-17T11:07:25.9199038Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9199316Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9199524Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9199731Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9200228Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9200442Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9200651Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9200655Z 
2025-07-17T11:07:25.9200658Z 
2025-07-17T11:07:25.9200753Z SIGABRT(6), PID: 1794249, Thread 1794514: 
2025-07-17T11:07:25.9201223Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9201437Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9201642Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9201845Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9202340Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9202562Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9202769Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9202868Z SIGABRT(6), PID: 1794247, Thread 1794319: 
2025-07-17T11:07:25.9203333Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9203551Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9203764Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9203963Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9204515Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9204807Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9205033Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9205039Z 
2025-07-17T11:07:25.9205042Z 
2025-07-17T11:07:25.9205138Z SIGABRT(6), PID: 1794246, Thread 1794274: 
2025-07-17T11:07:25.9205677Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9205894Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9206108Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9206364Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9206870Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9207087Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9207367Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9207371Z 
2025-07-17T11:07:25.9207464Z SIGABRT(6), PID: 1794252, Thread 1799069: 
2025-07-17T11:07:25.9207939Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9208147Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9208322Z frame #2: __poll + 0x4f (0x7fe18ccd5bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9208545Z frame #3: <unknown function> + 0xd65a2 (0x7fe0510eb5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9208764Z frame #4: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9208973Z frame #5: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9209078Z SIGABRT(6), PID: 1794250, Thread 1794509: 
2025-07-17T11:07:25.9209549Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9209767Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9209971Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9210176Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9210675Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9210892Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9211104Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9211110Z 
2025-07-17T11:07:25.9211113Z 
2025-07-17T11:07:25.9211211Z SIGABRT(6), PID: 1794251, Thread 1794541: 
2025-07-17T11:07:25.9211676Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9211888Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9212098Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9212303Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9212802Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9213087Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9213299Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9213400Z SIGABRT(6), PID: 1794252, Thread 1799078: 
2025-07-17T11:07:25.9213936Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9214157Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9214363Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9214628Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9214850Z frame #4: <unknown function> + 0xd1e7f (0x7fe0510e6e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9215067Z frame #5: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9215338Z frame #6: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9215349Z 
2025-07-17T11:07:25.9215353Z 
2025-07-17T11:07:25.9215449Z SIGABRT(6), PID: 1794247, Thread 1794321: 
2025-07-17T11:07:25.9215975Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9216185Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9216397Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9216598Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9217103Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9217313Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9217532Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9217628Z SIGABRT(6), PID: 1794249, Thread 1794516: 
2025-07-17T11:07:25.9218120Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9218330Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9218547Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9218751Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9219269Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9219480Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9219698Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9219702Z 
2025-07-17T11:07:25.9219707Z 
2025-07-17T11:07:25.9219796Z SIGABRT(6), PID: 1794246, Thread 1794275: 
2025-07-17T11:07:25.9220281Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9220493Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9220709Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9220998Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9221513Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9221726Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9222012Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9222017Z 
2025-07-17T11:07:25.9222112Z SIGABRT(6), PID: 1794250, Thread 1794511: 
2025-07-17T11:07:25.9222665Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9222876Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9223092Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9223293Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9223921Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9224143Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9224364Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9224368Z 
2025-07-17T11:07:25.9224463Z SIGABRT(6), PID: 1794251, Thread 1794544: 
2025-07-17T11:07:25.9224946Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9225162Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9225377Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9225575Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9226091Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9226303Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9226515Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9226519Z 
2025-07-17T11:07:25.9226618Z SIGABRT(6), PID: 1794247, Thread 1794323: 
2025-07-17T11:07:25.9227090Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9227308Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9227519Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9227730Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9228232Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9228447Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9228661Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9228665Z 
2025-07-17T11:07:25.9228763Z SIGABRT(6), PID: 1794252, Thread 1799087: 
2025-07-17T11:07:25.9229237Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9229530Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9229742Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9229955Z frame #3: <unknown function> + 0x9ca53 (0x7fe18cc59a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9230272Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x558910b8cbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9230509Z frame #5: <unknown function> + 0x1d314f (0x558910c4b14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9230735Z frame #6: <unknown function> + 0x1d2f6b (0x558910c4af6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9231020Z frame #7: <unknown function> + 0x6df36 (0x558910ae5f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9231269Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9231505Z frame #9: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9231746Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9232037Z frame #11: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9232275Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9232507Z frame #13: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9232743Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9232977Z frame #15: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9233212Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9233441Z frame #17: <unknown function> + 0x147734 (0x558910bbf734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9233664Z frame #18: <unknown function> + 0x21ca82 (0x558910c94a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9233896Z frame #19: <unknown function> + 0x21c9e4 (0x558910c949e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9234113Z frame #20: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9234334Z frame #21: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9234339Z 
2025-07-17T11:07:25.9234432Z SIGABRT(6), PID: 1794249, Thread 1794518: 
2025-07-17T11:07:25.9234913Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9235124Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9235343Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9235550Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9236060Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9236276Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9236496Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9236501Z 
2025-07-17T11:07:25.9236593Z SIGABRT(6), PID: 1794246, Thread 1794276: 
2025-07-17T11:07:25.9237077Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9237376Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9237593Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9237800Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9238367Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9238581Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9238803Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9238807Z 
2025-07-17T11:07:25.9238957Z SIGABRT(6), PID: 1794250, Thread 1794513: 
2025-07-17T11:07:25.9239441Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9239654Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9239939Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9240144Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9240660Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9240875Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9241098Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9241102Z 
2025-07-17T11:07:25.9241197Z SIGABRT(6), PID: 1794251, Thread 1794547: 
2025-07-17T11:07:25.9241676Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9241886Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9242094Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9242303Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9242798Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9243011Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9243223Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9243229Z 
2025-07-17T11:07:25.9243329Z SIGABRT(6), PID: 1794247, Thread 1794325: 
2025-07-17T11:07:25.9243793Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9244006Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9244211Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9244418Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9244913Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9245129Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9245346Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9245518Z SIGABRT(6), PID: 1794249, Thread 1794520: 
2025-07-17T11:07:25.9245985Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9246202Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9246467Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9246678Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9247176Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9247450Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9247668Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9247671Z 
2025-07-17T11:07:25.9247675Z 
2025-07-17T11:07:25.9247774Z SIGABRT(6), PID: 1794246, Thread 1794277: 
2025-07-17T11:07:25.9248306Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9248524Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9248734Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9248943Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9249445Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9249663Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9249877Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9249881Z 
2025-07-17T11:07:25.9249984Z SIGABRT(6), PID: 1794252, Thread 1799103: 
2025-07-17T11:07:25.9250456Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9250677Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9250847Z frame #2: read + 0x4c (0x7fe18ccd181c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9251036Z frame #3: _Py_read + 0x4c (0x558910c52a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9251265Z frame #4: <unknown function> + 0x208c3b (0x558910c80c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9251493Z frame #5: <unknown function> + 0x14648d (0x558910bbe48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9251743Z frame #6: PyObject_VectorcallMethod + 0x85 (0x558910bcd215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9251970Z frame #7: <unknown function> + 0x208ae7 (0x558910c80ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9252187Z frame #8: <unknown function> + 0x208472 (0x558910c80472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9252416Z frame #9: <unknown function> + 0x1da6ee (0x558910c526ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9252644Z frame #10: <unknown function> + 0x1483bc (0x558910bc03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9252894Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9253123Z frame #12: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9253369Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x558910ba43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9253664Z frame #14: <unknown function> + 0x147734 (0x558910bbf734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9253918Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x558910ba6c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9254147Z frame #16: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9254451Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9254678Z frame #18: _PyFunction_Vectorcall + 0x6c (0x558910bb40ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9254920Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x558910ba47b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9255220Z frame #20: <unknown function> + 0x147734 (0x558910bbf734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9255451Z frame #21: <unknown function> + 0x21ca82 (0x558910c94a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9255683Z frame #22: <unknown function> + 0x21c9e4 (0x558910c949e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9255967Z frame #23: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9256274Z frame #24: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9256278Z 
2025-07-17T11:07:25.9256382Z SIGABRT(6), PID: 1794250, Thread 1794515: 
2025-07-17T11:07:25.9256856Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9257078Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9257294Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9257507Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9258007Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9258230Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9258443Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9258455Z 
2025-07-17T11:07:25.9258552Z SIGABRT(6), PID: 1794251, Thread 1794551: 
2025-07-17T11:07:25.9259030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9259242Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9259461Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9259665Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9260166Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9260376Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9260591Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9260595Z 
2025-07-17T11:07:25.9260687Z SIGABRT(6), PID: 1794247, Thread 1794327: 
2025-07-17T11:07:25.9261156Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9261368Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9261584Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9261870Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9262377Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9262590Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9262874Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9262879Z 
2025-07-17T11:07:25.9262975Z SIGABRT(6), PID: 1794249, Thread 1794522: 
2025-07-17T11:07:25.9263517Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9263730Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9263945Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9264143Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9264702Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9264913Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9265129Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9265134Z 
2025-07-17T11:07:25.9265227Z SIGABRT(6), PID: 1794246, Thread 1794279: 
2025-07-17T11:07:25.9265707Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9265921Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9266132Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9266334Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9266842Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9267053Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9267274Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9267278Z 
2025-07-17T11:07:25.9267370Z SIGABRT(6), PID: 1794250, Thread 1794517: 
2025-07-17T11:07:25.9267845Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9268055Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9268268Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9268472Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9268970Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9269181Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9269393Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9269397Z 
2025-07-17T11:07:25.9269493Z SIGABRT(6), PID: 1794251, Thread 1794554: 
2025-07-17T11:07:25.9269957Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9270238Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9270446Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9270649Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9271209Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9271424Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9271690Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9271694Z 
2025-07-17T11:07:25.9271793Z SIGABRT(6), PID: 1794247, Thread 1794329: 
2025-07-17T11:07:25.9272258Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9272470Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9272734Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9272940Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9273433Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9273646Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9273859Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9273865Z 
2025-07-17T11:07:25.9273960Z SIGABRT(6), PID: 1794249, Thread 1794524: 
2025-07-17T11:07:25.9274424Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9274638Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9274844Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9275053Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9275546Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9275759Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9275968Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9275974Z 
2025-07-17T11:07:25.9276072Z SIGABRT(6), PID: 1794246, Thread 1794280: 
2025-07-17T11:07:25.9276537Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9276751Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9276961Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9277167Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9277665Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9277878Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9278160Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9278164Z 
2025-07-17T11:07:25.9278260Z SIGABRT(6), PID: 1794250, Thread 1794519: 
2025-07-17T11:07:25.9278727Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9279104Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9279321Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9279521Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9280082Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9280298Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9280515Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9280519Z 
2025-07-17T11:07:25.9280671Z SIGABRT(6), PID: 1794251, Thread 1794557: 
2025-07-17T11:07:25.9281147Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9281356Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9281566Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9281763Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9282262Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9282470Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9282683Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9282775Z SIGABRT(6), PID: 1794246, Thread 1794281: 
2025-07-17T11:07:25.9283252Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9283460Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9283670Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9283871Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9284373Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9284584Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9284799Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9284891Z SIGABRT(6), PID: 1794249, Thread 1794526: 
2025-07-17T11:07:25.9285364Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9285571Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9285786Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9285987Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9286493Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9286771Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9286992Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9286996Z 
2025-07-17T11:07:25.9287090Z SIGABRT(6), PID: 1794247, Thread 1794331: 
2025-07-17T11:07:25.9287640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9287854Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9288121Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9288326Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9288828Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9289096Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9289314Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9289319Z 
2025-07-17T11:07:25.9289322Z 
2025-07-17T11:07:25.9289327Z 
2025-07-17T11:07:25.9289418Z SIGABRT(6), PID: 1794250, Thread 1794521: 
2025-07-17T11:07:25.9289891Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9290102Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9290308Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9290516Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9291013Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9291230Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9291441Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9291444Z 
2025-07-17T11:07:25.9291540Z SIGABRT(6), PID: 1794251, Thread 1794560: 
2025-07-17T11:07:25.9292005Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9292219Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9292426Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9292629Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9293121Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9293335Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9293542Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9293637Z SIGABRT(6), PID: 1794246, Thread 1794282: 
2025-07-17T11:07:25.9294103Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9294315Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9294604Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9294812Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9295305Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9295577Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9295790Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9295885Z SIGABRT(6), PID: 1794247, Thread 1794333: 
2025-07-17T11:07:25.9296480Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9296698Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9296899Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9297110Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9297674Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9297890Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9298100Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9298104Z 
2025-07-17T11:07:25.9298108Z 
2025-07-17T11:07:25.9298111Z 
2025-07-17T11:07:25.9298207Z SIGABRT(6), PID: 1794249, Thread 1794527: 
2025-07-17T11:07:25.9298673Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9298888Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9299094Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9299299Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9299793Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9300006Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9300218Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9300230Z 
2025-07-17T11:07:25.9300323Z SIGABRT(6), PID: 1794250, Thread 1794523: 
2025-07-17T11:07:25.9300795Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9301001Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9301218Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9301419Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9301925Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9302136Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9302352Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9302436Z 
2025-07-17T11:07:25.9302529Z SIGABRT(6), PID: 1794249, Thread 1794529: 
2025-07-17T11:07:25.9303003Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9303212Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9303423Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9303688Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9304196Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9304461Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9304685Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9304781Z SIGABRT(6), PID: 1794251, Thread 1794563: 
2025-07-17T11:07:25.9305254Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9305520Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9305735Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9305933Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9306432Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9306641Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9306858Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9306949Z SIGABRT(6), PID: 1794247, Thread 1794334: 
2025-07-17T11:07:25.9307422Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9307630Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9307841Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9308038Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9308537Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9308749Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9308969Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9309060Z SIGABRT(6), PID: 1794246, Thread 1794283: 
2025-07-17T11:07:25.9309535Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9309745Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9309962Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9310162Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9310666Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9310875Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9311161Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9311166Z 
2025-07-17T11:07:25.9311169Z 
2025-07-17T11:07:25.9311174Z 
2025-07-17T11:07:25.9311177Z 
2025-07-17T11:07:25.9311268Z SIGABRT(6), PID: 1794250, Thread 1794525: 
2025-07-17T11:07:25.9311800Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9312015Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9312224Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9312490Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9312988Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9313209Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9313496Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9313500Z 
2025-07-17T11:07:25.9313597Z SIGABRT(6), PID: 1794249, Thread 1794532: 
2025-07-17T11:07:25.9314069Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9314282Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9314488Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9314695Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9315193Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9315405Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9315618Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9315622Z 
2025-07-17T11:07:25.9315717Z SIGABRT(6), PID: 1794251, Thread 1794566: 
2025-07-17T11:07:25.9316183Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9316396Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9316602Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9316805Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9317299Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9317512Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9317722Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9317820Z SIGABRT(6), PID: 1794247, Thread 1794337: 
2025-07-17T11:07:25.9318283Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9318493Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9318702Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9318998Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9319496Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9319709Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9319976Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9319981Z 
2025-07-17T11:07:25.9319986Z 
2025-07-17T11:07:25.9320083Z SIGABRT(6), PID: 1794246, Thread 1794284: 
2025-07-17T11:07:25.9320552Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9320823Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9321035Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9321242Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9321734Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9322006Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9322219Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9322227Z 
2025-07-17T11:07:25.9322317Z SIGABRT(6), PID: 1794250, Thread 1794528: 
2025-07-17T11:07:25.9322791Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9322999Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9323209Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9323409Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9323910Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9324122Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9324338Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9324342Z 
2025-07-17T11:07:25.9324432Z SIGABRT(6), PID: 1794252, Thread 1799160: 
2025-07-17T11:07:25.9324907Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9325116Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9325325Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9325524Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9325907Z frame #4: <unknown function> + 0x5a02023 (0x7fe0d107f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9326484Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe0d10831ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9327089Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe0d107a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9327539Z frame #7: <unknown function> + 0x79d655 (0x7fe0e256a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9327778Z frame #8: <unknown function> + 0xdc253 (0x7fe0995b5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9327993Z frame #9: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9328219Z frame #10: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9328279Z 
2025-07-17T11:07:25.9328375Z SIGABRT(6), PID: 1794249, Thread 1794535: 
2025-07-17T11:07:25.9328854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9329122Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9329338Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9329540Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9330044Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9330312Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9330530Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9330534Z 
2025-07-17T11:07:25.9330624Z SIGABRT(6), PID: 1794250, Thread 1794531: 
2025-07-17T11:07:25.9331096Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9331306Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9331519Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9331722Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9332228Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9332439Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9332658Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9332663Z 
2025-07-17T11:07:25.9332759Z SIGABRT(6), PID: 1794246, Thread 1794285: 
2025-07-17T11:07:25.9333233Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9333449Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9333659Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9333868Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9334370Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9334591Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9334802Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9334806Z 
2025-07-17T11:07:25.9334901Z SIGABRT(6), PID: 1794247, Thread 1794339: 
2025-07-17T11:07:25.9335367Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9335653Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9335857Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9336118Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9336687Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9336907Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9337121Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9337126Z 
2025-07-17T11:07:25.9337297Z SIGABRT(6), PID: 1794251, Thread 1794569: 
2025-07-17T11:07:25.9337782Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9338007Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9338296Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9338504Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9339009Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9339223Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9339431Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9339437Z 
2025-07-17T11:07:25.9339535Z SIGABRT(6), PID: 1794249, Thread 1794538: 
2025-07-17T11:07:25.9340003Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9340216Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9340424Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9340629Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9341132Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9341344Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9341556Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9341653Z SIGABRT(6), PID: 1794250, Thread 1794533: 
2025-07-17T11:07:25.9342120Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9342336Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9342545Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9342752Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9343248Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9343463Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9343672Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9343762Z 
2025-07-17T11:07:25.9343766Z 
2025-07-17T11:07:25.9343860Z SIGABRT(6), PID: 1794247, Thread 1794341: 
2025-07-17T11:07:25.9344334Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9344542Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9344809Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9345013Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9345513Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9345777Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9346003Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9346094Z SIGABRT(6), PID: 1794246, Thread 1794286: 
2025-07-17T11:07:25.9346565Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9346856Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9347070Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9347271Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9347774Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9347985Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9348202Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9348293Z SIGABRT(6), PID: 1794251, Thread 1794572: 
2025-07-17T11:07:25.9348768Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9348977Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9349189Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9349387Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9349888Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9350096Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9350311Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9350314Z 
2025-07-17T11:07:25.9350319Z 
2025-07-17T11:07:25.9350323Z 
2025-07-17T11:07:25.9350413Z SIGABRT(6), PID: 1794249, Thread 1794540: 
2025-07-17T11:07:25.9350889Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9351096Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9351307Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9351506Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9352006Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9352285Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9352507Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9352511Z 
2025-07-17T11:07:25.9352604Z SIGABRT(6), PID: 1794250, Thread 1794536: 
2025-07-17T11:07:25.9353132Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9353343Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9353555Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9353814Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9354319Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9354534Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9354806Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9354811Z 
2025-07-17T11:07:25.9354907Z SIGABRT(6), PID: 1794246, Thread 1794288: 
2025-07-17T11:07:25.9355377Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9362148Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9362448Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9362684Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9363222Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9363452Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9363671Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9363780Z SIGABRT(6), PID: 1794247, Thread 1794343: 
2025-07-17T11:07:25.9364279Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9364502Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9364710Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9364920Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9365427Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9365643Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9365857Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9365955Z SIGABRT(6), PID: 1794251, Thread 1794575: 
2025-07-17T11:07:25.9366433Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9366649Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9366852Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9367180Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9367686Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9367901Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9368171Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9368177Z 
2025-07-17T11:07:25.9368181Z 
2025-07-17T11:07:25.9368185Z 
2025-07-17T11:07:25.9368288Z SIGABRT(6), PID: 1794249, Thread 1794543: 
2025-07-17T11:07:25.9368815Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9369034Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9369241Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9369446Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9370020Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9370235Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9370443Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9370447Z 
2025-07-17T11:07:25.9370543Z SIGABRT(6), PID: 1794251, Thread 1794578: 
2025-07-17T11:07:25.9371018Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9371231Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9371434Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9371636Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9372134Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9372342Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9372551Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9372558Z 
2025-07-17T11:07:25.9372649Z SIGABRT(6), PID: 1794250, Thread 1794539: 
2025-07-17T11:07:25.9373123Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9373332Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9373538Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9373738Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9374241Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9374452Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9374665Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9374756Z SIGABRT(6), PID: 1794246, Thread 1794290: 
2025-07-17T11:07:25.9375235Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9375514Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9375723Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9375984Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9376558Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9376773Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9377056Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9377061Z 
2025-07-17T11:07:25.9377065Z 
2025-07-17T11:07:25.9377161Z SIGABRT(6), PID: 1794247, Thread 1794345: 
2025-07-17T11:07:25.9377638Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9377923Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9378133Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9378333Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9378836Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9379043Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9379257Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9379263Z 
2025-07-17T11:07:25.9379352Z SIGABRT(6), PID: 1794249, Thread 1794546: 
2025-07-17T11:07:25.9379825Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9380031Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9380241Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9380440Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9380941Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9381152Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9381364Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9381367Z 
2025-07-17T11:07:25.9381456Z SIGABRT(6), PID: 1794251, Thread 1794581: 
2025-07-17T11:07:25.9381925Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9382131Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9382340Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9382535Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9383037Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9383249Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9383542Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9383546Z 
2025-07-17T11:07:25.9383639Z SIGABRT(6), PID: 1794246, Thread 1794291: 
2025-07-17T11:07:25.9384110Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9384381Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9384591Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9384793Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9385387Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9385608Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9385819Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9385973Z SIGABRT(6), PID: 1794250, Thread 1794542: 
2025-07-17T11:07:25.9386447Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9386665Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9386872Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9387077Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9387576Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9387789Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9388002Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9388097Z SIGABRT(6), PID: 1794247, Thread 1794347: 
2025-07-17T11:07:25.9388563Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9388774Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9388978Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9389184Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9389679Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9389891Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9390101Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9390106Z 
2025-07-17T11:07:25.9390109Z 
2025-07-17T11:07:25.9390112Z 
2025-07-17T11:07:25.9390206Z SIGABRT(6), PID: 1794249, Thread 1794549: 
2025-07-17T11:07:25.9390670Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9390878Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9391081Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9391283Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9391843Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9392055Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9392260Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9392264Z 
2025-07-17T11:07:25.9392413Z SIGABRT(6), PID: 1794251, Thread 1794585: 
2025-07-17T11:07:25.9392884Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9393095Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9393360Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9393563Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9394065Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9394332Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9394546Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9394551Z 
2025-07-17T11:07:25.9394641Z SIGABRT(6), PID: 1794250, Thread 1794545: 
2025-07-17T11:07:25.9395108Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9395317Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9395526Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9395722Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9396219Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9396428Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9396639Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9396643Z 
2025-07-17T11:07:25.9396731Z SIGABRT(6), PID: 1794252, Thread 1799161: 
2025-07-17T11:07:25.9397202Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9397408Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9397615Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9397812Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9398198Z frame #4: <unknown function> + 0x5a02023 (0x7fe0d107f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9398772Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe0d10831ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9399374Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe0d107a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9399762Z frame #7: <unknown function> + 0x79d655 (0x7fe0e256a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9400075Z frame #8: <unknown function> + 0xdc253 (0x7fe0995b5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9400284Z frame #9: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9400508Z frame #10: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9400514Z 
2025-07-17T11:07:25.9400604Z SIGABRT(6), PID: 1794246, Thread 1794293: 
2025-07-17T11:07:25.9401140Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9401357Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9401643Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9401849Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9402356Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9402625Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9402842Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9402846Z 
2025-07-17T11:07:25.9402939Z SIGABRT(6), PID: 1794247, Thread 1794349: 
2025-07-17T11:07:25.9403409Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9403621Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9403823Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9404026Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9404523Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9404735Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9404946Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9404951Z 
2025-07-17T11:07:25.9405043Z SIGABRT(6), PID: 1794249, Thread 1794552: 
2025-07-17T11:07:25.9405507Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9405716Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9405921Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9406121Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9406619Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9406831Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9407038Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9407042Z 
2025-07-17T11:07:25.9407134Z SIGABRT(6), PID: 1794251, Thread 1794588: 
2025-07-17T11:07:25.9407597Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9407808Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9408086Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9408291Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9408786Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9409058Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9409270Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9409274Z 
2025-07-17T11:07:25.9409368Z SIGABRT(6), PID: 1794250, Thread 1794548: 
2025-07-17T11:07:25.9409894Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9410108Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9410314Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9410575Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9411077Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9411292Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9411502Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9411506Z 
2025-07-17T11:07:25.9411602Z SIGABRT(6), PID: 1794247, Thread 1794351: 
2025-07-17T11:07:25.9412066Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9412277Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9412482Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9412685Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9413177Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9413388Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9413600Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9413609Z 
2025-07-17T11:07:25.9413700Z SIGABRT(6), PID: 1794246, Thread 1794295: 
2025-07-17T11:07:25.9414175Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9414381Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9414591Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9414788Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9415287Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9415492Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9415704Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9415708Z 
2025-07-17T11:07:25.9416016Z SIGABRT(6), PID: 1794249, Thread 1794555: 
2025-07-17T11:07:25.9416487Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9416697Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9416903Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9417195Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9417722Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9418003Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9418220Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9418227Z 
2025-07-17T11:07:25.9418317Z SIGABRT(6), PID: 1794251, Thread 1794590: 
2025-07-17T11:07:25.9418789Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9419072Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9419283Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9419483Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9419978Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9420191Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9420407Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9420411Z 
2025-07-17T11:07:25.9420500Z SIGABRT(6), PID: 1794247, Thread 1794354: 
2025-07-17T11:07:25.9420976Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9421183Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9421391Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9421591Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9422092Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9422299Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9422514Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9422519Z 
2025-07-17T11:07:25.9422609Z SIGABRT(6), PID: 1794250, Thread 1794550: 
2025-07-17T11:07:25.9423085Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9423294Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9423502Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9423700Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9424200Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9424495Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9424709Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9424715Z 
2025-07-17T11:07:25.9424805Z SIGABRT(6), PID: 1794246, Thread 1794297: 
2025-07-17T11:07:25.9425333Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9425548Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9425754Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9425957Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9426510Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9426728Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9426937Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9426998Z 
2025-07-17T11:07:25.9427093Z SIGABRT(6), PID: 1794249, Thread 1794558: 
2025-07-17T11:07:25.9427566Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9427771Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9427975Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9428177Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9428670Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9428880Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9429088Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9429092Z 
2025-07-17T11:07:25.9429185Z SIGABRT(6), PID: 1794251, Thread 1794592: 
2025-07-17T11:07:25.9429653Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9429864Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9430068Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9430267Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9430761Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9430970Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9431177Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9431181Z 
2025-07-17T11:07:25.9431274Z SIGABRT(6), PID: 1794250, Thread 1794553: 
2025-07-17T11:07:25.9431738Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9431949Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9432155Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9432424Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9432912Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9433124Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9433389Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9433394Z 
2025-07-17T11:07:25.9433490Z SIGABRT(6), PID: 1794249, Thread 1794561: 
2025-07-17T11:07:25.9433955Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9434222Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9434433Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9434632Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9435123Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9435391Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9435601Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9435695Z SIGABRT(6), PID: 1794247, Thread 1794357: 
2025-07-17T11:07:25.9436166Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9436374Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9436585Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9436780Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9437284Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9437493Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9437706Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9437711Z 
2025-07-17T11:07:25.9437715Z 
2025-07-17T11:07:25.9437803Z SIGABRT(6), PID: 1794246, Thread 1794299: 
2025-07-17T11:07:25.9438274Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9438485Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9438696Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9438897Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9439395Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9439603Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9439816Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9439820Z 
2025-07-17T11:07:25.9439910Z SIGABRT(6), PID: 1794251, Thread 1794595: 
2025-07-17T11:07:25.9440380Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9440654Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9440860Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9441061Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9441617Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9441824Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9442036Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9442040Z 
2025-07-17T11:07:25.9442184Z SIGABRT(6), PID: 1794250, Thread 1794556: 
2025-07-17T11:07:25.9442661Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9442866Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9443135Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9443333Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9443836Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9444044Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9444257Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9444261Z 
2025-07-17T11:07:25.9444353Z SIGABRT(6), PID: 1794249, Thread 1794564: 
2025-07-17T11:07:25.9444826Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9445032Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9445241Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9445438Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9445936Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9446142Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9446355Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9446360Z 
2025-07-17T11:07:25.9446453Z SIGABRT(6), PID: 1794246, Thread 1794301: 
2025-07-17T11:07:25.9446917Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9447134Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9447338Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9447541Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9448039Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9448253Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9448558Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9448562Z 
2025-07-17T11:07:25.9448655Z SIGABRT(6), PID: 1794247, Thread 1794360: 
2025-07-17T11:07:25.9449120Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9449332Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9449593Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9449798Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9450352Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9450569Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9450782Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9450874Z SIGABRT(6), PID: 1794251, Thread 1794598: 
2025-07-17T11:07:25.9451406Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9451617Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9451821Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9452021Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9452515Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9452725Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9452930Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9453024Z SIGABRT(6), PID: 1794250, Thread 1794559: 
2025-07-17T11:07:25.9453492Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9453702Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9453906Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9454107Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9454598Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9454813Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9455020Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9455126Z SIGABRT(6), PID: 1794249, Thread 1794567: 
2025-07-17T11:07:25.9455597Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9455817Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9456075Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9456280Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9456779Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9457081Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9457292Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9457393Z SIGABRT(6), PID: 1794246, Thread 1794303: 
2025-07-17T11:07:25.9457936Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9458160Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9458369Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9458649Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9459155Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9459376Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9459597Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9459771Z SIGABRT(6), PID: 1794252, Thread 1799162: 
2025-07-17T11:07:25.9460243Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9460459Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9460665Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9460873Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9461250Z frame #4: <unknown function> + 0x5a02023 (0x7fe0d107f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9461830Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe0d10831ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9462435Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe0d107a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9462818Z frame #7: <unknown function> + 0x79d655 (0x7fe0e256a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9463063Z frame #8: <unknown function> + 0xdc253 (0x7fe0995b5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9463273Z frame #9: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9463502Z frame #10: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9463507Z 
2025-07-17T11:07:25.9463510Z 
2025-07-17T11:07:25.9463513Z 
2025-07-17T11:07:25.9463516Z 
2025-07-17T11:07:25.9463521Z 
2025-07-17T11:07:25.9463524Z 
2025-07-17T11:07:25.9463617Z SIGABRT(6), PID: 1794250, Thread 1794562: 
2025-07-17T11:07:25.9464104Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9464316Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9464531Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9464730Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9465243Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9465529Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9465750Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9465756Z 
2025-07-17T11:07:25.9465849Z SIGABRT(6), PID: 1794249, Thread 1794570: 
2025-07-17T11:07:25.9466394Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9466613Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9466819Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9467074Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9467582Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9467800Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9468068Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9468172Z SIGABRT(6), PID: 1794247, Thread 1794362: 
2025-07-17T11:07:25.9468646Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9468873Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9469080Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9469292Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9469799Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9470019Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9470235Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9470339Z SIGABRT(6), PID: 1794251, Thread 1794601: 
2025-07-17T11:07:25.9470814Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9471027Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9471233Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9471446Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9471945Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9472160Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9472370Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9472374Z 
2025-07-17T11:07:25.9472380Z 
2025-07-17T11:07:25.9472482Z SIGABRT(6), PID: 1794246, Thread 1794305: 
2025-07-17T11:07:25.9472953Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9473169Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9473379Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9473660Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9474156Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9474376Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9474646Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9474651Z 
2025-07-17T11:07:25.9474654Z 
2025-07-17T11:07:25.9474758Z SIGABRT(6), PID: 1794250, Thread 1794565: 
2025-07-17T11:07:25.9475284Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9475503Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9475714Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9475929Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9476507Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9476728Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9476937Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9476941Z 
2025-07-17T11:07:25.9477038Z SIGABRT(6), PID: 1794247, Thread 1794364: 
2025-07-17T11:07:25.9477516Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9477731Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9477943Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9478142Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9478642Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9478851Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9479073Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9479078Z 
2025-07-17T11:07:25.9479169Z SIGABRT(6), PID: 1794249, Thread 1794573: 
2025-07-17T11:07:25.9479642Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9479852Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9480061Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9480263Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9480772Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9480980Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9481198Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9481202Z 
2025-07-17T11:07:25.9481294Z SIGABRT(6), PID: 1794251, Thread 1794604: 
2025-07-17T11:07:25.9481765Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9482044Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9482256Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9482452Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9483028Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9483242Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9483517Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9483609Z SIGABRT(6), PID: 1794246, Thread 1794306: 
2025-07-17T11:07:25.9484088Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9484297Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9484576Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9484780Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9485286Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9485495Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9485713Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9485720Z 
2025-07-17T11:07:25.9485723Z 
2025-07-17T11:07:25.9485814Z SIGABRT(6), PID: 1794250, Thread 1794568: 
2025-07-17T11:07:25.9486287Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9486497Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9486709Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9486909Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9487409Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9487621Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9487839Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9487842Z 
2025-07-17T11:07:25.9487936Z SIGABRT(6), PID: 1794249, Thread 1794576: 
2025-07-17T11:07:25.9488411Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9488629Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9488838Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9489045Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9489540Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9489753Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9490030Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9490132Z SIGABRT(6), PID: 1794247, Thread 1794366: 
2025-07-17T11:07:25.9490603Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9490818Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9491079Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9491291Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9491838Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9492058Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9492273Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9492277Z 
2025-07-17T11:07:25.9492337Z 
2025-07-17T11:07:25.9492439Z SIGABRT(6), PID: 1794251, Thread 1794607: 
2025-07-17T11:07:25.9492906Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9493122Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9493328Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9493536Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9494032Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9494248Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9494461Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9494467Z 
2025-07-17T11:07:25.9494571Z SIGABRT(6), PID: 1794246, Thread 1794308: 
2025-07-17T11:07:25.9495041Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9495257Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9495468Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9495675Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9496218Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9496439Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9496654Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9496658Z 
2025-07-17T11:07:25.9496756Z SIGABRT(6), PID: 1794249, Thread 1794579: 
2025-07-17T11:07:25.9497222Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9497436Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9497640Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9497843Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9498420Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9498638Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9498853Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9498869Z 
2025-07-17T11:07:25.9499031Z SIGABRT(6), PID: 1794250, Thread 1794571: 
2025-07-17T11:07:25.9499511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9499724Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9500003Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9500208Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9500711Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9500995Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9501216Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9501222Z 
2025-07-17T11:07:25.9501317Z SIGABRT(6), PID: 1794251, Thread 1794611: 
2025-07-17T11:07:25.9501791Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9501998Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9502207Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9502408Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9502909Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9503126Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9503341Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9503345Z 
2025-07-17T11:07:25.9503436Z SIGABRT(6), PID: 1794247, Thread 1794368: 
2025-07-17T11:07:25.9503911Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9504122Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9504340Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9504539Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9505045Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9505258Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9505476Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9505480Z 
2025-07-17T11:07:25.9505574Z SIGABRT(6), PID: 1794246, Thread 1794310: 
2025-07-17T11:07:25.9506049Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9506330Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9506544Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9506743Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9507249Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9507514Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9507737Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9507742Z 
2025-07-17T11:07:25.9507836Z SIGABRT(6), PID: 1794249, Thread 1794582: 
2025-07-17T11:07:25.9508383Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9508594Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9508802Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9509060Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9509570Z frame #4: <unknown function> + 0x3475ab (0x7f457fb6c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9509778Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9509995Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9509999Z 
2025-07-17T11:07:25.9510091Z SIGABRT(6), PID: 1794250, Thread 1794574: 
2025-07-17T11:07:25.9510565Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9510783Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9510991Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9511195Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9511690Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9511907Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9512118Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9512122Z 
2025-07-17T11:07:25.9512220Z SIGABRT(6), PID: 1794251, Thread 1794613: 
2025-07-17T11:07:25.9512692Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9512910Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9513116Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9513326Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9513819Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9514030Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9514239Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9514404Z SIGABRT(6), PID: 1794247, Thread 1794370: 
2025-07-17T11:07:25.9514868Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9515081Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9515284Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9515546Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9516043Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9516314Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9516528Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9516535Z 
2025-07-17T11:07:25.9516539Z 
2025-07-17T11:07:25.9516634Z SIGABRT(6), PID: 1794246, Thread 1794312: 
2025-07-17T11:07:25.9517100Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9517376Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9517587Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9517792Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9518296Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9518516Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9518730Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9518833Z SIGABRT(6), PID: 1794249, Thread 1795310: 
2025-07-17T11:07:25.9519302Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9519765Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f44913cb08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9519972Z frame #2: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9520176Z frame #3: pthread_kill + 0x12c (0x7f4582cd69fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9520348Z frame #4: raise + 0x16 (0x7f4582c82476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9520514Z frame #5: abort + 0xd3 (0x7f4582c687f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9520760Z frame #6: <unknown function> + 0x972cc (0x7f43fef542cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9520999Z frame #7: <unknown function> + 0x95f47 (0x7f43fef52f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9521231Z frame #8: <unknown function> + 0x34b71 (0x7f43feef1b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9521450Z frame #9: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9521671Z frame #10: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9521676Z 
2025-07-17T11:07:25.9521690Z 
2025-07-17T11:07:25.9521786Z SIGABRT(6), PID: 1794250, Thread 1794577: 
2025-07-17T11:07:25.9522272Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9522488Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9522776Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9522978Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9523494Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9523761Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9523988Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9523993Z 
2025-07-17T11:07:25.9524091Z SIGABRT(6), PID: 1794251, Thread 1794615: 
2025-07-17T11:07:25.9524630Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9524845Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9525061Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9525318Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9525827Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9526035Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9526257Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9526261Z 
2025-07-17T11:07:25.9526355Z SIGABRT(6), PID: 1794252, Thread 1799163: 
2025-07-17T11:07:25.9526833Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9527047Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9527255Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9527456Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9527848Z frame #4: <unknown function> + 0x5a02023 (0x7fe0d107f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9528425Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe0d10831ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9529027Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe0d107a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9529417Z frame #7: <unknown function> + 0x79d655 (0x7fe0e256a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9529663Z frame #8: <unknown function> + 0xdc253 (0x7fe0995b5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9529874Z frame #9: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9530101Z frame #10: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9530105Z 
2025-07-17T11:07:25.9530198Z SIGABRT(6), PID: 1794247, Thread 1794372: 
2025-07-17T11:07:25.9530677Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9530890Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9531173Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9531378Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9531893Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9532162Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9532388Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9532393Z 
2025-07-17T11:07:25.9532486Z SIGABRT(6), PID: 1794246, Thread 1794314: 
2025-07-17T11:07:25.9533015Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9533238Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9533449Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9533655Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9534217Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9534437Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9534650Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9534654Z 
2025-07-17T11:07:25.9534755Z SIGABRT(6), PID: 1794249, Thread 1798838: 
2025-07-17T11:07:25.9535224Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9535441Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9535610Z frame #2: __poll + 0x4f (0x7f4582d58bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9536077Z frame #3: <unknown function> + 0x489f (0x7f458286489f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.9536311Z frame #4: <unknown function> + 0x1483bc (0x555dfb2113bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9536573Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9536803Z frame #6: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9537053Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9537276Z frame #8: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9537535Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x555dfb1f9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9537768Z frame #10: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9538023Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x555dfb1f7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9538251Z frame #12: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9538500Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9538725Z frame #14: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9538971Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9539194Z frame #16: <unknown function> + 0x147734 (0x555dfb210734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9539506Z frame #17: <unknown function> + 0x21ca82 (0x555dfb2e5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9539729Z frame #18: <unknown function> + 0x21c9e4 (0x555dfb2e59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9539954Z frame #19: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9540166Z frame #20: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9540171Z 
2025-07-17T11:07:25.9540364Z SIGABRT(6), PID: 1794250, Thread 1794580: 
2025-07-17T11:07:25.9540844Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9541131Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9541348Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9541562Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9542067Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9542463Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9542681Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9542685Z 
2025-07-17T11:07:25.9542783Z SIGABRT(6), PID: 1794251, Thread 1794617: 
2025-07-17T11:07:25.9543256Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9543475Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9543686Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9543898Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9544396Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9544618Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9544835Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9544839Z 
2025-07-17T11:07:25.9544940Z SIGABRT(6), PID: 1794247, Thread 1794374: 
2025-07-17T11:07:25.9545407Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9545628Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9545841Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9546052Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9546551Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9546772Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9546985Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9547000Z 
2025-07-17T11:07:25.9547092Z SIGABRT(6), PID: 1794246, Thread 1794316: 
2025-07-17T11:07:25.9547572Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9547852Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9548070Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9548276Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9548845Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9549060Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9549286Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9549290Z 
2025-07-17T11:07:25.9549439Z SIGABRT(6), PID: 1794250, Thread 1794584: 
2025-07-17T11:07:25.9549920Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9550136Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9550353Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9550618Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9551127Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9551336Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9551559Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9551565Z 
2025-07-17T11:07:25.9551657Z SIGABRT(6), PID: 1794251, Thread 1794619: 
2025-07-17T11:07:25.9552134Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9552344Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9552565Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9552768Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9553280Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9553488Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9553712Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9553716Z 
2025-07-17T11:07:25.9553812Z SIGABRT(6), PID: 1794247, Thread 1794376: 
2025-07-17T11:07:25.9554288Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9554498Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9554709Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9554913Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9555421Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9555636Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9555857Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9555928Z 
2025-07-17T11:07:25.9556023Z SIGABRT(6), PID: 1794246, Thread 1794318: 
2025-07-17T11:07:25.9556508Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9556725Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9556999Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9557204Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9557712Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9557980Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9558211Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9558215Z 
2025-07-17T11:07:25.9558310Z SIGABRT(6), PID: 1794250, Thread 1794587: 
2025-07-17T11:07:25.9558785Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9559065Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9559277Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9559486Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9559983Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9560204Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9560420Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9560423Z 
2025-07-17T11:07:25.9560524Z SIGABRT(6), PID: 1794251, Thread 1794621: 
2025-07-17T11:07:25.9560992Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9561209Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9561417Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9561625Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9562126Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9562343Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9562553Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9562558Z 
2025-07-17T11:07:25.9562655Z SIGABRT(6), PID: 1794247, Thread 1794378: 
2025-07-17T11:07:25.9563122Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9563343Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9563548Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9563754Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9564250Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9564559Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9564774Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9564780Z 
2025-07-17T11:07:25.9564878Z SIGABRT(6), PID: 1794249, Thread 1798839: 
2025-07-17T11:07:25.9565402Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9565626Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9565831Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9566122Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f4582cd42dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9566575Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f44bd113048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9567012Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f44bd115bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9567307Z frame #6: <unknown function> + 0xdc253 (0x7f448f637253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9567525Z frame #7: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9567737Z frame #8: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9567741Z 
2025-07-17T11:07:25.9567843Z SIGABRT(6), PID: 1794246, Thread 1794320: 
2025-07-17T11:07:25.9568314Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9568538Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9568749Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9568959Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9569462Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9569683Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9569899Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9569913Z 
2025-07-17T11:07:25.9570005Z SIGABRT(6), PID: 1794250, Thread 1794589: 
2025-07-17T11:07:25.9570485Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9570697Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9570912Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9571116Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9571629Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9571841Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9572062Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9572066Z 
2025-07-17T11:07:25.9572161Z SIGABRT(6), PID: 1794251, Thread 1794623: 
2025-07-17T11:07:25.9572645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9572947Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9573165Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9573369Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9573933Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9574147Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9574425Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9574429Z 
2025-07-17T11:07:25.9574524Z SIGABRT(6), PID: 1794247, Thread 1794380: 
2025-07-17T11:07:25.9575001Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9575214Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9575497Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9575699Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9576259Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9576471Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9576694Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9576702Z 
2025-07-17T11:07:25.9576793Z SIGABRT(6), PID: 1794246, Thread 1794322: 
2025-07-17T11:07:25.9577274Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9577489Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9577712Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9577917Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9578428Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9578642Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9578861Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9578957Z SIGABRT(6), PID: 1794251, Thread 1794625: 
2025-07-17T11:07:25.9579432Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9579642Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9579857Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9580058Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9580561Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9580773Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9581072Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9581166Z SIGABRT(6), PID: 1794250, Thread 1794591: 
2025-07-17T11:07:25.9581647Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9581859Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9582143Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9582347Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9582925Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9583147Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9583361Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9583455Z SIGABRT(6), PID: 1794247, Thread 1794382: 
2025-07-17T11:07:25.9584002Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9584220Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9584425Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9584630Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9585126Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9585348Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9585562Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9585664Z SIGABRT(6), PID: 1794249, Thread 1798840: 
2025-07-17T11:07:25.9586129Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9586345Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9586550Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9586785Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f4582cd42dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9587264Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f44bd10d741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9587510Z frame #5: <unknown function> + 0xdc253 (0x7f448f637253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9587722Z frame #6: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9587941Z frame #7: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9587945Z 
2025-07-17T11:07:25.9587948Z 
2025-07-17T11:07:25.9587951Z 
2025-07-17T11:07:25.9587955Z 
2025-07-17T11:07:25.9587957Z 
2025-07-17T11:07:25.9588052Z SIGABRT(6), PID: 1794252, Thread 1799164: 
2025-07-17T11:07:25.9588534Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9588748Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9588964Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9589235Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9589621Z frame #4: <unknown function> + 0x5a02023 (0x7fe0d107f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9590191Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe0d10831ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9590853Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe0d107a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9591311Z frame #7: <unknown function> + 0x79d655 (0x7fe0e256a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9591553Z frame #8: <unknown function> + 0xdc253 (0x7fe0995b5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9591774Z frame #9: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9591994Z frame #10: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9592056Z 
2025-07-17T11:07:25.9592159Z SIGABRT(6), PID: 1794249, Thread 1798841: 
2025-07-17T11:07:25.9592637Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9592856Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9593028Z frame #2: ioctl + 0x3f (0x7f4582d5a94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9593281Z frame #3: <unknown function> + 0x13eef0 (0x7f43feffbef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9593514Z frame #4: <unknown function> + 0x1378c4 (0x7f43feff48c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9593757Z frame #5: <unknown function> + 0x95b2b (0x7f43fef52b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9593989Z frame #6: <unknown function> + 0x34b71 (0x7f43feef1b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9594208Z frame #7: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9594421Z frame #8: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9594427Z 
2025-07-17T11:07:25.9594529Z SIGABRT(6), PID: 1794247, Thread 1794384: 
2025-07-17T11:07:25.9594999Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9595218Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9595424Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9595639Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9596149Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9596376Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9596593Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9596699Z SIGABRT(6), PID: 1794250, Thread 1794594: 
2025-07-17T11:07:25.9597173Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9597394Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9597676Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9597887Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9598384Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9598610Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9598882Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9598985Z SIGABRT(6), PID: 1794251, Thread 1794628: 
2025-07-17T11:07:25.9599508Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9599728Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9599942Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9600153Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9600730Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9600952Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9601165Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9601169Z 
2025-07-17T11:07:25.9601174Z 
2025-07-17T11:07:25.9601279Z SIGABRT(6), PID: 1794246, Thread 1794324: 
2025-07-17T11:07:25.9601747Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9601966Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9602176Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9602390Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9602886Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9603103Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9603317Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9603321Z 
2025-07-17T11:07:25.9603332Z 
2025-07-17T11:07:25.9603427Z SIGABRT(6), PID: 1794247, Thread 1794386: 
2025-07-17T11:07:25.9603899Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9604108Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9604323Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9604525Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9605033Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9605242Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9605466Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9605471Z 
2025-07-17T11:07:25.9605566Z SIGABRT(6), PID: 1794249, Thread 1799057: 
2025-07-17T11:07:25.9606105Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9606314Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9606496Z frame #2: __poll + 0x4f (0x7f4582d58bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9606711Z frame #3: <unknown function> + 0xd4540 (0x7f444716b540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9606984Z frame #4: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9607199Z frame #5: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9607204Z 
2025-07-17T11:07:25.9607304Z SIGABRT(6), PID: 1794251, Thread 1794631: 
2025-07-17T11:07:25.9607829Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9608051Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9608256Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9608529Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9609031Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9609251Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9609460Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9609464Z 
2025-07-17T11:07:25.9609571Z SIGABRT(6), PID: 1794246, Thread 1794326: 
2025-07-17T11:07:25.9610039Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9610257Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9610471Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9610674Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9611176Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9611401Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9611616Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9611620Z 
2025-07-17T11:07:25.9611720Z SIGABRT(6), PID: 1794250, Thread 1794597: 
2025-07-17T11:07:25.9612186Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9612403Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9612617Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9612831Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9613333Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9613549Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9613764Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9613843Z 
2025-07-17T11:07:25.9613938Z SIGABRT(6), PID: 1794249, Thread 1799065: 
2025-07-17T11:07:25.9614411Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9614622Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9614798Z frame #2: __poll + 0x4f (0x7f4582d58bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9615073Z frame #3: <unknown function> + 0xd65a2 (0x7f444716d5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9615297Z frame #4: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9615511Z frame #5: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9615515Z 
2025-07-17T11:07:25.9615676Z SIGABRT(6), PID: 1794247, Thread 1794388: 
2025-07-17T11:07:25.9616206Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9616427Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9616709Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9616922Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9617419Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9617638Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9617852Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9617857Z 
2025-07-17T11:07:25.9617957Z SIGABRT(6), PID: 1794250, Thread 1794599: 
2025-07-17T11:07:25.9618430Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9618650Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9618861Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9619074Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9619571Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9619791Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9620012Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9620017Z 
2025-07-17T11:07:25.9620116Z SIGABRT(6), PID: 1794246, Thread 1794328: 
2025-07-17T11:07:25.9620585Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9620809Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9621019Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9621242Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9621741Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9621963Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9622260Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9622264Z 
2025-07-17T11:07:25.9622363Z SIGABRT(6), PID: 1794251, Thread 1794634: 
2025-07-17T11:07:25.9622832Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9623055Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9623332Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9623546Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9624111Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9624338Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9624556Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9624560Z 
2025-07-17T11:07:25.9624659Z SIGABRT(6), PID: 1794249, Thread 1799077: 
2025-07-17T11:07:25.9625204Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9625433Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9625640Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9625852Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9626075Z frame #4: <unknown function> + 0xd1e7f (0x7f4447168e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9626297Z frame #5: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9626511Z frame #6: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9626522Z 
2025-07-17T11:07:25.9626614Z SIGABRT(6), PID: 1794247, Thread 1794390: 
2025-07-17T11:07:25.9627097Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9627311Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9627522Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9627727Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9628240Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9628453Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9628676Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9628680Z 
2025-07-17T11:07:25.9628775Z SIGABRT(6), PID: 1794250, Thread 1794603: 
2025-07-17T11:07:25.9629251Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9629463Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9629680Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9629883Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9630392Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9630687Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9630910Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9630915Z 
2025-07-17T11:07:25.9631008Z SIGABRT(6), PID: 1794246, Thread 1794330: 
2025-07-17T11:07:25.9631543Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9631764Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9631980Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9632261Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9632769Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9632985Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9633264Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9633268Z 
2025-07-17T11:07:25.9633361Z SIGABRT(6), PID: 1794251, Thread 1794638: 
2025-07-17T11:07:25.9633837Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9634046Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9634262Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9634468Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9634981Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9635191Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9635412Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9635416Z 
2025-07-17T11:07:25.9635510Z SIGABRT(6), PID: 1794249, Thread 1799081: 
2025-07-17T11:07:25.9635988Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9636195Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9636415Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9636629Z frame #3: <unknown function> + 0x9ca53 (0x7f4582cdca53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9636897Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x555dfb1ddbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9637133Z frame #5: <unknown function> + 0x1d314f (0x555dfb29c14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9637375Z frame #6: <unknown function> + 0x1d2f6b (0x555dfb29bf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9637603Z frame #7: <unknown function> + 0x6df36 (0x555dfb136f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9637865Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9638095Z frame #9: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9638358Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9638590Z frame #11: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9638914Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9639147Z frame #13: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9639401Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9639684Z frame #15: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9639933Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9640158Z frame #17: <unknown function> + 0x147734 (0x555dfb210734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9640449Z frame #18: <unknown function> + 0x21ca82 (0x555dfb2e5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9640675Z frame #19: <unknown function> + 0x21c9e4 (0x555dfb2e59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9640901Z frame #20: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9641118Z frame #21: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9641197Z 
2025-07-17T11:07:25.9641303Z SIGABRT(6), PID: 1794247, Thread 1794392: 
2025-07-17T11:07:25.9641781Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9642015Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9642238Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9642445Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9642964Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9643180Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9643406Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9643412Z 
2025-07-17T11:07:25.9643505Z SIGABRT(6), PID: 1794250, Thread 1794606: 
2025-07-17T11:07:25.9643992Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9644203Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9644420Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9644630Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9645139Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9645352Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9645573Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9645577Z 
2025-07-17T11:07:25.9645671Z SIGABRT(6), PID: 1794246, Thread 1794332: 
2025-07-17T11:07:25.9646159Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9646373Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9646593Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9646868Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9647381Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9647595Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9647815Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9647878Z 
2025-07-17T11:07:25.9647974Z SIGABRT(6), PID: 1794251, Thread 1794641: 
2025-07-17T11:07:25.9648451Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9648720Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9648945Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9649154Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9649659Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9649931Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9650156Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9650160Z 
2025-07-17T11:07:25.9650252Z SIGABRT(6), PID: 1794247, Thread 1794394: 
2025-07-17T11:07:25.9650726Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9650938Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9651157Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9651359Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9651866Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9652075Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9652294Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9652298Z 
2025-07-17T11:07:25.9652394Z SIGABRT(6), PID: 1794249, Thread 1799093: 
2025-07-17T11:07:25.9652876Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9653088Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9653258Z frame #2: read + 0x4c (0x7f4582d5481c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9653456Z frame #3: _Py_read + 0x4c (0x555dfb2a3a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9653686Z frame #4: <unknown function> + 0x208c3b (0x555dfb2d1c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9653913Z frame #5: <unknown function> + 0x14648d (0x555dfb20f48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9654175Z frame #6: PyObject_VectorcallMethod + 0x85 (0x555dfb21e215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9654398Z frame #7: <unknown function> + 0x208ae7 (0x555dfb2d1ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9654625Z frame #8: <unknown function> + 0x208472 (0x555dfb2d1472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9654850Z frame #9: <unknown function> + 0x1da6ee (0x555dfb2a36ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9655156Z frame #10: <unknown function> + 0x1483bc (0x555dfb2113bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9655403Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9655643Z frame #12: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9655884Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x555dfb1f53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9656253Z frame #14: <unknown function> + 0x147734 (0x555dfb210734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9656513Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x555dfb1f7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9656745Z frame #16: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9657055Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9657295Z frame #18: _PyFunction_Vectorcall + 0x6c (0x555dfb2050ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9657531Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x555dfb1f57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9657832Z frame #20: <unknown function> + 0x147734 (0x555dfb210734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9658054Z frame #21: <unknown function> + 0x21ca82 (0x555dfb2e5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9658284Z frame #22: <unknown function> + 0x21c9e4 (0x555dfb2e59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9658500Z frame #23: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9658723Z frame #24: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9658727Z 
2025-07-17T11:07:25.9658827Z SIGABRT(6), PID: 1794250, Thread 1794609: 
2025-07-17T11:07:25.9659314Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9659540Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9659759Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9659968Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9660474Z frame #4: <unknown function> + 0x3475ab (0x7fcc70e7a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9660691Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9660907Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9660911Z 
2025-07-17T11:07:25.9661011Z SIGABRT(6), PID: 1794251, Thread 1794644: 
2025-07-17T11:07:25.9661487Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9661706Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9661915Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9662129Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9662628Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9662842Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9663057Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9663138Z 
2025-07-17T11:07:25.9663244Z SIGABRT(6), PID: 1794252, Thread 1799165: 
2025-07-17T11:07:25.9663719Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9663940Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9664151Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9664435Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9664822Z frame #4: <unknown function> + 0x5a02023 (0x7fe0d107f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9665468Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe0d10831ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9666074Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe0d107a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9666530Z frame #7: <unknown function> + 0x79d655 (0x7fe0e256a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9666767Z frame #8: <unknown function> + 0xdc253 (0x7fe0995b5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9666996Z frame #9: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9667216Z frame #10: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9667323Z SIGABRT(6), PID: 1794246, Thread 1794335: 
2025-07-17T11:07:25.9667803Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9668026Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9668241Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9668453Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9668960Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9669182Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9669397Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9669401Z 
2025-07-17T11:07:25.9669507Z SIGABRT(6), PID: 1794247, Thread 1794396: 
2025-07-17T11:07:25.9669984Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9670194Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9670411Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9670610Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9671110Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9671322Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9671543Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9671547Z 
2025-07-17T11:07:25.9671550Z 
2025-07-17T11:07:25.9671715Z SIGABRT(6), PID: 1794250, Thread 1795308: 
2025-07-17T11:07:25.9672197Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9672655Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fcb826da08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9672928Z frame #2: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9673130Z frame #3: pthread_kill + 0x12c (0x7fcc73fe49fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9673305Z frame #4: raise + 0x16 (0x7fcc73f90476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9673468Z frame #5: abort + 0xd3 (0x7fcc73f767f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9673887Z frame #6: <unknown function> + 0x972cc (0x7fcaf02632cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9674134Z frame #7: <unknown function> + 0x95f47 (0x7fcaf0261f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9674376Z frame #8: <unknown function> + 0x34b71 (0x7fcaf0200b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9674593Z frame #9: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9674879Z frame #10: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9674883Z 
2025-07-17T11:07:25.9674983Z SIGABRT(6), PID: 1794251, Thread 1794647: 
2025-07-17T11:07:25.9675469Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9675688Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9675905Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9676113Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9676621Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9676831Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9677049Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9677055Z 
2025-07-17T11:07:25.9677146Z SIGABRT(6), PID: 1794247, Thread 1794398: 
2025-07-17T11:07:25.9677624Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9677834Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9678046Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9678250Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9678756Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9678970Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9679197Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9679200Z 
2025-07-17T11:07:25.9679293Z SIGABRT(6), PID: 1794246, Thread 1794336: 
2025-07-17T11:07:25.9679776Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9679991Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9680277Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9680481Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9680991Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9681260Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9681484Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9681489Z 
2025-07-17T11:07:25.9681580Z SIGABRT(6), PID: 1794250, Thread 1798031: 
2025-07-17T11:07:25.9682124Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9682346Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9682517Z frame #2: __poll + 0x4f (0x7fcc74066bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9682940Z frame #3: <unknown function> + 0x489f (0x7fcc73b7289f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.9683228Z frame #4: <unknown function> + 0x1483bc (0x55c8f76493bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9683483Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9683717Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9683965Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9684198Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9684450Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55c8f7631a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9684685Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9684946Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c8f762fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9685175Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9685425Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9685652Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9685899Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9686124Z frame #16: <unknown function> + 0x147734 (0x55c8f7648734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9686357Z frame #17: <unknown function> + 0x21ca82 (0x55c8f771da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9686579Z frame #18: <unknown function> + 0x21c9e4 (0x55c8f771d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9686803Z frame #19: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9687020Z frame #20: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9687024Z 
2025-07-17T11:07:25.9687127Z SIGABRT(6), PID: 1794251, Thread 1794650: 
2025-07-17T11:07:25.9687604Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9687828Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9688043Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9688327Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9688831Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9689059Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9689277Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9689341Z 
2025-07-17T11:07:25.9689447Z SIGABRT(6), PID: 1794247, Thread 1794400: 
2025-07-17T11:07:25.9689921Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9690201Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9690414Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9690625Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9691126Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9691413Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9691632Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9691637Z 
2025-07-17T11:07:25.9691738Z SIGABRT(6), PID: 1794246, Thread 1794338: 
2025-07-17T11:07:25.9692215Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9692438Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9692650Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9692865Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9693370Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9693593Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9693810Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9693814Z 
2025-07-17T11:07:25.9693917Z SIGABRT(6), PID: 1794251, Thread 1794653: 
2025-07-17T11:07:25.9694387Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9694609Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9694816Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9695024Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9695518Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9695738Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9696019Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9696024Z 
2025-07-17T11:07:25.9696129Z SIGABRT(6), PID: 1794247, Thread 1794402: 
2025-07-17T11:07:25.9696599Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9696932Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9697147Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9697350Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9697919Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9698133Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9698355Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9698359Z 
2025-07-17T11:07:25.9698521Z SIGABRT(6), PID: 1794246, Thread 1794340: 
2025-07-17T11:07:25.9699001Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9699220Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9699512Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9699713Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9700217Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9700431Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9700649Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9700744Z SIGABRT(6), PID: 1794251, Thread 1794656: 
2025-07-17T11:07:25.9701222Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9701429Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9701642Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9701843Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9702347Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9702557Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9702777Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9702782Z 
2025-07-17T11:07:25.9702787Z 
2025-07-17T11:07:25.9702882Z SIGABRT(6), PID: 1794247, Thread 1794404: 
2025-07-17T11:07:25.9703356Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9703567Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9703777Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9703981Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9704492Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9704703Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9704926Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9704999Z 
2025-07-17T11:07:25.9705097Z SIGABRT(6), PID: 1794251, Thread 1794659: 
2025-07-17T11:07:25.9705578Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9705788Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9706065Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9706265Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9706769Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9707037Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9707266Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9707269Z 
2025-07-17T11:07:25.9707364Z SIGABRT(6), PID: 1794246, Thread 1794342: 
2025-07-17T11:07:25.9707838Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9708118Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9708329Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9708538Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9709040Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9709263Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9709477Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9709578Z SIGABRT(6), PID: 1794250, Thread 1798033: 
2025-07-17T11:07:25.9710050Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9710269Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9710478Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9710716Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fcc73fe22dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9711163Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fcbae422048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9711598Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fcbae424bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9711834Z frame #6: <unknown function> + 0xdc253 (0x7fcb80946253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9712064Z frame #7: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9712276Z frame #8: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9712283Z 
2025-07-17T11:07:25.9712286Z 
2025-07-17T11:07:25.9712387Z SIGABRT(6), PID: 1794247, Thread 1794406: 
2025-07-17T11:07:25.9712862Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9713087Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9713364Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9713578Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9714091Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9714317Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9714590Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9714595Z 
2025-07-17T11:07:25.9714701Z SIGABRT(6), PID: 1794251, Thread 1794662: 
2025-07-17T11:07:25.9715232Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9715452Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9715662Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9715867Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9716424Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9716642Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9716855Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9716859Z 
2025-07-17T11:07:25.9716962Z SIGABRT(6), PID: 1794249, Thread 1799120: 
2025-07-17T11:07:25.9717440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9717659Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9717863Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9718068Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9718442Z frame #4: <unknown function> + 0x5a02023 (0x7f44c7101023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9719022Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f44c71051ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9719624Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f44c70fc1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9720009Z frame #7: <unknown function> + 0x79d655 (0x7f44d85ec655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9720249Z frame #8: <unknown function> + 0xdc253 (0x7f448f637253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9720462Z frame #9: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9720690Z frame #10: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9720695Z 
2025-07-17T11:07:25.9720786Z SIGABRT(6), PID: 1794246, Thread 1794344: 
2025-07-17T11:07:25.9721272Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9721485Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9721702Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9721978Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9722490Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9722706Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9722983Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9722988Z 
2025-07-17T11:07:25.9723082Z SIGABRT(6), PID: 1794247, Thread 1794408: 
2025-07-17T11:07:25.9723558Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9723846Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9724065Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9724270Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9724855Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9725072Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9725291Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9725296Z 
2025-07-17T11:07:25.9725390Z SIGABRT(6), PID: 1794250, Thread 1798034: 
2025-07-17T11:07:25.9725870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9726080Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9726301Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9726527Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fcc73fe22dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9727019Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fcbae41c741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9727252Z frame #5: <unknown function> + 0xdc253 (0x7fcb80946253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9727474Z frame #6: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9727684Z frame #7: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9727688Z 
2025-07-17T11:07:25.9727792Z SIGABRT(6), PID: 1794251, Thread 1794665: 
2025-07-17T11:07:25.9728264Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9728480Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9728688Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9728899Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9729398Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9729616Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9729827Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9729840Z 
2025-07-17T11:07:25.9730004Z SIGABRT(6), PID: 1794246, Thread 1794346: 
2025-07-17T11:07:25.9730489Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9730705Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9730922Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9731180Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9731693Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9731906Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9732181Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9732281Z SIGABRT(6), PID: 1794252, Thread 1799166: 
2025-07-17T11:07:25.9732753Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9733024Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9733244Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9733446Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9733833Z frame #4: <unknown function> + 0x5a02023 (0x7fe0d107f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9734402Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe0d10831ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9735012Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe0d107a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9735401Z frame #7: <unknown function> + 0x79d655 (0x7fe0e256a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9735643Z frame #8: <unknown function> + 0xdc253 (0x7fe0995b5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9735857Z frame #9: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9736137Z frame #10: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9736142Z 
2025-07-17T11:07:25.9736146Z 
2025-07-17T11:07:25.9736239Z SIGABRT(6), PID: 1794250, Thread 1798117: 
2025-07-17T11:07:25.9744153Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9744435Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9744631Z frame #2: ioctl + 0x3f (0x7fcc7406894f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9744891Z frame #3: <unknown function> + 0x13eef0 (0x7fcaf030aef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9745142Z frame #4: <unknown function> + 0x1378c4 (0x7fcaf03038c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9745381Z frame #5: <unknown function> + 0x95b2b (0x7fcaf0261b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9745620Z frame #6: <unknown function> + 0x34b71 (0x7fcaf0200b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9745844Z frame #7: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9746071Z frame #8: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9746315Z SIGABRT(6), PID: 1794247, Thread 1794410: 
2025-07-17T11:07:25.9746838Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9747066Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9747280Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9747570Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9748109Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9748397Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9748627Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9748633Z 
2025-07-17T11:07:25.9748637Z 
2025-07-17T11:07:25.9748735Z SIGABRT(6), PID: 1794251, Thread 1794668: 
2025-07-17T11:07:25.9749225Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9749531Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9749742Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9749958Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9750465Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9750681Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9750894Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9750899Z 
2025-07-17T11:07:25.9750999Z SIGABRT(6), PID: 1794246, Thread 1794348: 
2025-07-17T11:07:25.9751474Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9751694Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9751902Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9752111Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9752608Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9752828Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9753037Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9753043Z 
2025-07-17T11:07:25.9753143Z SIGABRT(6), PID: 1794247, Thread 1794412: 
2025-07-17T11:07:25.9753614Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9753827Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9754051Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9754764Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9755603Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9756504Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9757034Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9757357Z 
2025-07-17T11:07:25.9757454Z SIGABRT(6), PID: 1794250, Thread 1799056: 
2025-07-17T11:07:25.9758182Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9759062Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9759551Z frame #2: __poll + 0x4f (0x7fcc74066bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9760097Z frame #3: <unknown function> + 0xd4540 (0x7fcb3847a540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9760679Z frame #4: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9761213Z frame #5: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9761520Z 
2025-07-17T11:07:25.9761688Z SIGABRT(6), PID: 1794251, Thread 1794671: 
2025-07-17T11:07:25.9762360Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9763147Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9763703Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9764211Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9765013Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9765992Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9766510Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9766818Z 
2025-07-17T11:07:25.9766916Z SIGABRT(6), PID: 1794247, Thread 1794414: 
2025-07-17T11:07:25.9767567Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9768335Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9768838Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9769339Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9770139Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9770960Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9771486Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9771892Z SIGABRT(6), PID: 1794250, Thread 1799063: 
2025-07-17T11:07:25.9772553Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9773327Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9773806Z frame #2: __poll + 0x4f (0x7fcc74066bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9774291Z frame #3: <unknown function> + 0xd65a2 (0x7fcb3847c5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9774814Z frame #4: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9775426Z frame #5: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9775822Z SIGABRT(6), PID: 1794246, Thread 1794350: 
2025-07-17T11:07:25.9776542Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9777388Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9777906Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9778415Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9779315Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9780125Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9780642Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9781024Z 
2025-07-17T11:07:25.9781027Z 
2025-07-17T11:07:25.9781031Z 
2025-07-17T11:07:25.9781130Z SIGABRT(6), PID: 1794251, Thread 1794674: 
2025-07-17T11:07:25.9781783Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9782547Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9783057Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9783562Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9784356Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9785155Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9785670Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9785980Z 
2025-07-17T11:07:25.9786079Z SIGABRT(6), PID: 1794250, Thread 1799075: 
2025-07-17T11:07:25.9786735Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9787501Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9788009Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9788514Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9789033Z frame #4: <unknown function> + 0xd1e7f (0x7fcb38477e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9789559Z frame #5: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9790079Z frame #6: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9790389Z 
2025-07-17T11:07:25.9790483Z SIGABRT(6), PID: 1794247, Thread 1794416: 
2025-07-17T11:07:25.9791133Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9791900Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9792415Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9792918Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9793801Z frame #4: <unknown function> + 0x3475ab (0x7f566ad045ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9794598Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9795118Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9795429Z 
2025-07-17T11:07:25.9795520Z SIGABRT(6), PID: 1794246, Thread 1794353: 
2025-07-17T11:07:25.9796235Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9797008Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9797576Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9798088Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9798883Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9799739Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9800263Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9800572Z 
2025-07-17T11:07:25.9800671Z SIGABRT(6), PID: 1794251, Thread 1794677: 
2025-07-17T11:07:25.9801320Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9802084Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9802602Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9803111Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9803900Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9804706Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9805222Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9805527Z 
2025-07-17T11:07:25.9805617Z SIGABRT(6), PID: 1794247, Thread 1795312: 
2025-07-17T11:07:25.9806261Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9807267Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f557c56108a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9808010Z frame #2: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9808515Z frame #3: pthread_kill + 0x12c (0x7f566de6e9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9808977Z frame #4: raise + 0x16 (0x7f566de1a476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9809399Z frame #5: abort + 0xd3 (0x7f566de007f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9809900Z frame #6: <unknown function> + 0x972cc (0x7f54ea0ea2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9810470Z frame #7: <unknown function> + 0x95f47 (0x7f54ea0e8f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9811021Z frame #8: <unknown function> + 0x34b71 (0x7f54ea087b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9811559Z frame #9: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9812173Z frame #10: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9812489Z 
2025-07-17T11:07:25.9812589Z SIGABRT(6), PID: 1794250, Thread 1799086: 
2025-07-17T11:07:25.9813244Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9814013Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9814586Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9815106Z frame #3: <unknown function> + 0x9ca53 (0x7fcc73feaa53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9815671Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55c8f7615bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9816371Z frame #5: <unknown function> + 0x1d314f (0x55c8f76d414f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9816926Z frame #6: <unknown function> + 0x1d2f6b (0x55c8f76d3f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9817470Z frame #7: <unknown function> + 0x6df36 (0x55c8f756ef36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9818034Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9818718Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9819292Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9819858Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9820427Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9820995Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9821555Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9822119Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9822677Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9823231Z frame #17: <unknown function> + 0x147734 (0x55c8f7648734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9823773Z frame #18: <unknown function> + 0x21ca82 (0x55c8f771da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9824318Z frame #19: <unknown function> + 0x21c9e4 (0x55c8f771d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9824847Z frame #20: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9825379Z frame #21: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9825689Z 
2025-07-17T11:07:25.9825788Z SIGABRT(6), PID: 1794251, Thread 1794679: 
2025-07-17T11:07:25.9826444Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9827215Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9827733Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9828243Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9829046Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9829852Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9830378Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9830870Z SIGABRT(6), PID: 1794246, Thread 1794356: 
2025-07-17T11:07:25.9831529Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9832299Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9832807Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9833384Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9834182Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9835045Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9835571Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9835894Z 
2025-07-17T11:07:25.9835898Z 
2025-07-17T11:07:25.9835995Z SIGABRT(6), PID: 1794247, Thread 1799040: 
2025-07-17T11:07:25.9836653Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9837489Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9837970Z frame #2: __poll + 0x4f (0x7f566def0bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9838652Z frame #3: <unknown function> + 0x489f (0x7f566d9fc89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.9839383Z frame #4: <unknown function> + 0x1483bc (0x56305eab13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9839962Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9840533Z frame #6: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9841100Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9841667Z frame #8: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9842235Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x56305ea99a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9842815Z frame #10: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9843398Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x56305ea97c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9843972Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9844541Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9845112Z frame #14: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9845668Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9846231Z frame #16: <unknown function> + 0x147734 (0x56305eab0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9846765Z frame #17: <unknown function> + 0x21ca82 (0x56305eb85a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9847308Z frame #18: <unknown function> + 0x21c9e4 (0x56305eb859e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9847847Z frame #19: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9848380Z frame #20: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9848698Z 
2025-07-17T11:07:25.9848799Z SIGABRT(6), PID: 1794249, Thread 1799121: 
2025-07-17T11:07:25.9849462Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9850318Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9850837Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9851355Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9852120Z frame #4: <unknown function> + 0x5a02023 (0x7f44c7101023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9853163Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f44c71051ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9854465Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f44c70fc1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9855543Z frame #7: <unknown function> + 0x79d655 (0x7f44d85ec655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9856376Z frame #8: <unknown function> + 0xdc253 (0x7f448f637253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9856915Z frame #9: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9857438Z frame #10: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9857747Z 
2025-07-17T11:07:25.9857855Z SIGABRT(6), PID: 1794250, Thread 1799101: 
2025-07-17T11:07:25.9858517Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9859301Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9859783Z frame #2: read + 0x4c (0x7fcc7406281c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9860238Z frame #3: _Py_read + 0x4c (0x55c8f76dba6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9860754Z frame #4: <unknown function> + 0x208c3b (0x55c8f7709c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9861305Z frame #5: <unknown function> + 0x14648d (0x55c8f764748d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9861881Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55c8f7656215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9862449Z frame #7: <unknown function> + 0x208ae7 (0x55c8f7709ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9863001Z frame #8: <unknown function> + 0x208472 (0x55c8f7709472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9863556Z frame #9: <unknown function> + 0x1da6ee (0x55c8f76db6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9864103Z frame #10: <unknown function> + 0x1483bc (0x55c8f76493bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9864677Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9865265Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9865845Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55c8f762d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9866417Z frame #14: <unknown function> + 0x147734 (0x55c8f7648734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9866996Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c8f762fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9867586Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9868164Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9868823Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55c8f763d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9869391Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55c8f762d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9869956Z frame #20: <unknown function> + 0x147734 (0x55c8f7648734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9870500Z frame #21: <unknown function> + 0x21ca82 (0x55c8f771da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9871118Z frame #22: <unknown function> + 0x21c9e4 (0x55c8f771d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9871668Z frame #23: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9872204Z frame #24: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9872512Z 
2025-07-17T11:07:25.9872689Z SIGABRT(6), PID: 1794246, Thread 1794358: 
2025-07-17T11:07:25.9873356Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9874144Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9874747Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9875271Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9876094Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9876908Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9877437Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9877745Z 
2025-07-17T11:07:25.9877846Z SIGABRT(6), PID: 1794251, Thread 1794683: 
2025-07-17T11:07:25.9878510Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9879275Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9879794Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9880308Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9881114Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9881919Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9882445Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9882768Z 
2025-07-17T11:07:25.9882861Z SIGABRT(6), PID: 1794252, Thread 1799167: 
2025-07-17T11:07:25.9883523Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe09b348b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9884303Z frame #1: <unknown function> + 0x42520 (0x7fe18cbff520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9884828Z frame #2: <unknown function> + 0x91117 (0x7fe18cc4e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9885343Z frame #3: pthread_cond_wait + 0x211 (0x7fe18cc50a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9886026Z frame #4: <unknown function> + 0x5a02023 (0x7fe0d107f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9887063Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe0d10831ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9888399Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe0d107a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9889469Z frame #7: <unknown function> + 0x79d655 (0x7fe0e256a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9890386Z frame #8: <unknown function> + 0xdc253 (0x7fe0995b5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9890949Z frame #9: <unknown function> + 0x94ac3 (0x7fe18cc51ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9891485Z frame #10: <unknown function> + 0x126850 (0x7fe18cce3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9891804Z 
2025-07-17T11:07:25.9891904Z SIGABRT(6), PID: 1794246, Thread 1794361: 
2025-07-17T11:07:25.9892626Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9893424Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9893952Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9894533Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9895346Z frame #4: <unknown function> + 0x3475ab (0x7fcf98a995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9896214Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9896752Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9897061Z 
2025-07-17T11:07:25.9897167Z SIGABRT(6), PID: 1794251, Thread 1794687: 
2025-07-17T11:07:25.9897823Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9898594Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9899111Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9899616Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9900418Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9901223Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9901746Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9902053Z 
2025-07-17T11:07:25.9902150Z SIGABRT(6), PID: 1794246, Thread 1795302: 
2025-07-17T11:07:25.9902807Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9903820Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fceaa2f808a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9904582Z frame #2: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9905088Z frame #3: pthread_kill + 0x12c (0x7fcf9bc039fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9905552Z frame #4: raise + 0x16 (0x7fcf9bbaf476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9905982Z frame #5: abort + 0xd3 (0x7fcf9bb957f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9906491Z frame #6: <unknown function> + 0x972cc (0x7fce17e812cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9907066Z frame #7: <unknown function> + 0x95f47 (0x7fce17e7ff47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9907726Z frame #8: <unknown function> + 0x34b71 (0x7fce17e1eb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9908271Z frame #9: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9908809Z frame #10: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9909125Z 
2025-07-17T11:07:25.9909224Z SIGABRT(6), PID: 1794251, Thread 1794690: 
2025-07-17T11:07:25.9909963Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9910737Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9911327Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9911846Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9912652Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9913531Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9914055Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9914365Z 
2025-07-17T11:07:25.9914473Z SIGABRT(6), PID: 1794247, Thread 1799041: 
2025-07-17T11:07:25.9915134Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9915902Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9916426Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9916969Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f566de6c2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9917738Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f55a82a9048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9918705Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f55a82abbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9919467Z frame #6: <unknown function> + 0xdc253 (0x7f557a7cd253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9920012Z frame #7: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9920531Z frame #8: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9920847Z 
2025-07-17T11:07:25.9920945Z SIGABRT(6), PID: 1794246, Thread 1797628: 
2025-07-17T11:07:25.9921610Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9922383Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9922870Z frame #2: __poll + 0x4f (0x7fcf9bc85bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9923552Z frame #3: <unknown function> + 0x489f (0x7fcf9b79189f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:25.9924285Z frame #4: <unknown function> + 0x1483bc (0x55a9caf5a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9924866Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9925446Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9926014Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9926683Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9927262Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55a9caf42a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9927850Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9928497Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a9caf40c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9929085Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9929658Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9930291Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9930865Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9931430Z frame #16: <unknown function> + 0x147734 (0x55a9caf59734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9931983Z frame #17: <unknown function> + 0x21ca82 (0x55a9cb02ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9932588Z frame #18: <unknown function> + 0x21c9e4 (0x55a9cb02e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:25.9933135Z frame #19: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9933673Z frame #20: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9933980Z 
2025-07-17T11:07:25.9934084Z SIGABRT(6), PID: 1794251, Thread 1794693: 
2025-07-17T11:07:25.9934740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9935524Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9936114Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9936627Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9937435Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9938248Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9938775Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9939095Z 
2025-07-17T11:07:25.9939192Z SIGABRT(6), PID: 1794251, Thread 1794696: 
2025-07-17T11:07:25.9939854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9940634Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9941156Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9941667Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9942473Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9943275Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9943803Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9944130Z 
2025-07-17T11:07:25.9944225Z SIGABRT(6), PID: 1794247, Thread 1799042: 
2025-07-17T11:07:25.9944877Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9945753Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9946263Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9946801Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f566de6c2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9947673Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f55a82a3741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:25.9948478Z frame #5: <unknown function> + 0xdc253 (0x7f557a7cd253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9949023Z frame #6: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9949619Z frame #7: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9949958Z 
2025-07-17T11:07:25.9950064Z SIGABRT(6), PID: 1794247, Thread 1799045: 
2025-07-17T11:07:25.9950719Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9951562Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9952039Z frame #2: ioctl + 0x3f (0x7f566def294f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9952543Z frame #3: <unknown function> + 0x13eef0 (0x7f54ea191ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9953110Z frame #4: <unknown function> + 0x1378c4 (0x7f54ea18a8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9953665Z frame #5: <unknown function> + 0x95b2b (0x7f54ea0e8b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9954227Z frame #6: <unknown function> + 0x34b71 (0x7f54ea087b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:25.9954768Z frame #7: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9955294Z frame #8: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9955608Z 
2025-07-17T11:07:25.9955705Z SIGABRT(6), PID: 1794251, Thread 1794699: 
2025-07-17T11:07:25.9956363Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9957127Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9957645Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9958188Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9958995Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9959802Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9960322Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9960638Z 
2025-07-17T11:07:25.9960731Z SIGABRT(6), PID: 1794251, Thread 1794702: 
2025-07-17T11:07:25.9961383Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9962149Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9962660Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9963172Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9963967Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9964843Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9965362Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9965667Z 
2025-07-17T11:07:25.9965771Z SIGABRT(6), PID: 1794247, Thread 1799060: 
2025-07-17T11:07:25.9966484Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9967253Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9967730Z frame #2: __poll + 0x4f (0x7f566def0bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9968270Z frame #3: <unknown function> + 0xd4540 (0x7f5532301540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9968811Z frame #4: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9969337Z frame #5: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9969704Z 
2025-07-17T11:07:25.9969808Z SIGABRT(6), PID: 1794249, Thread 1799122: 
2025-07-17T11:07:25.9970452Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9971217Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9971732Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9972240Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9972919Z frame #4: <unknown function> + 0x5a02023 (0x7f44c7101023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9973958Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f44c71051ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9975208Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f44c70fc1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9976330Z frame #7: <unknown function> + 0x79d655 (0x7f44d85ec655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9977039Z frame #8: <unknown function> + 0xdc253 (0x7f448f637253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9977587Z frame #9: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9978114Z frame #10: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9978429Z 
2025-07-17T11:07:25.9978526Z SIGABRT(6), PID: 1794250, Thread 1799144: 
2025-07-17T11:07:25.9979178Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9979948Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9980464Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9980973Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9981643Z frame #4: <unknown function> + 0x5a02023 (0x7fcbb8410023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9982671Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcbb84141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9984000Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcbb840b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:25.9985067Z frame #7: <unknown function> + 0x79d655 (0x7fcbc98fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:25.9985848Z frame #8: <unknown function> + 0xdc253 (0x7fcb80946253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:25.9986415Z frame #9: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9986946Z frame #10: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9987271Z 
2025-07-17T11:07:25.9987450Z SIGABRT(6), PID: 1794247, Thread 1799067: 
2025-07-17T11:07:25.9988124Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9988902Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9989482Z frame #2: __poll + 0x4f (0x7f566def0bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9989972Z frame #3: <unknown function> + 0xd65a2 (0x7f55323035a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9990504Z frame #4: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9991027Z frame #5: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9991423Z SIGABRT(6), PID: 1794251, Thread 1794705: 
2025-07-17T11:07:25.9992088Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9992869Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9993386Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9993895Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9994703Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:25.9995499Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9996024Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9996342Z 
2025-07-17T11:07:25.9996346Z 
2025-07-17T11:07:25.9996439Z SIGABRT(6), PID: 1794247, Thread 1799073: 
2025-07-17T11:07:25.9997098Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:25.9997871Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9998380Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9998885Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:25.9999402Z frame #4: <unknown function> + 0xd1e7f (0x7f55322fee7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:25.9999924Z frame #5: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0000449Z frame #6: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0000759Z 
2025-07-17T11:07:26.0000854Z SIGABRT(6), PID: 1794251, Thread 1794708: 
2025-07-17T11:07:26.0001505Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0002347Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0002863Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0003371Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0004233Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0005035Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0005556Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0005862Z 
2025-07-17T11:07:26.0006019Z SIGABRT(6), PID: 1794251, Thread 1794711: 
2025-07-17T11:07:26.0006677Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0007448Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0008021Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0008524Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0009320Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0010112Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0010635Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0010941Z 
2025-07-17T11:07:26.0011041Z SIGABRT(6), PID: 1794247, Thread 1799082: 
2025-07-17T11:07:26.0011690Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0012450Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0012961Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0013471Z frame #3: <unknown function> + 0x9ca53 (0x7f566de74a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0014029Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x56305ea7dbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0014607Z frame #5: <unknown function> + 0x1d314f (0x56305eb3c14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0015156Z frame #6: <unknown function> + 0x1d2f6b (0x56305eb3bf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0015699Z frame #7: <unknown function> + 0x6df36 (0x56305e9d6f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0016319Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0016887Z frame #9: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0017456Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0018032Z frame #11: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0018608Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0019172Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0019736Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0020306Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0020959Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0021521Z frame #17: <unknown function> + 0x147734 (0x56305eab0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0022066Z frame #18: <unknown function> + 0x21ca82 (0x56305eb85a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0022610Z frame #19: <unknown function> + 0x21c9e4 (0x56305eb859e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0023222Z frame #20: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0023768Z frame #21: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0024081Z 
2025-07-17T11:07:26.0024186Z SIGABRT(6), PID: 1794246, Thread 1797629: 
2025-07-17T11:07:26.0024927Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0025742Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0026253Z frame #2: epoll_wait + 0x5e (0x7fcf9bc92e2e in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0026940Z frame #3: uv__io_poll + 0x6bd (0x7fceec84e44d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0027679Z frame #4: uv_run + 0x12a (0x7fceec84236a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0028471Z frame #5: <unknown function> + 0x658a08a (0x7fcee0bb608a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0029172Z frame #6: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0029722Z frame #7: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0030257Z frame #8: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0030577Z 
2025-07-17T11:07:26.0030674Z SIGABRT(6), PID: 1794251, Thread 1794714: 
2025-07-17T11:07:26.0031338Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0032121Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0032640Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0033157Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0033961Z frame #4: <unknown function> + 0x3475ab (0x7f6041c9a5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0034770Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0035292Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0035603Z 
2025-07-17T11:07:26.0035697Z SIGABRT(6), PID: 1794251, Thread 1795314: 
2025-07-17T11:07:26.0036352Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0037371Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f5f534f808a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0038123Z frame #2: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0038626Z frame #3: pthread_kill + 0x12c (0x7f6044e049fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0039083Z frame #4: raise + 0x16 (0x7f6044db0476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0039585Z frame #5: abort + 0xd3 (0x7f6044d967f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0040081Z frame #6: <unknown function> + 0x972cc (0x7f5ec10812cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0040643Z frame #7: <unknown function> + 0x95f47 (0x7f5ec107ff47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0041205Z frame #8: <unknown function> + 0x34b71 (0x7f5ec101eb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0041822Z frame #9: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0042352Z frame #10: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0042659Z 
2025-07-17T11:07:26.0042763Z SIGABRT(6), PID: 1794247, Thread 1799091: 
2025-07-17T11:07:26.0043481Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0044260Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0044728Z frame #2: read + 0x4c (0x7f566deec81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0045178Z frame #3: _Py_read + 0x4c (0x56305eb43a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0045740Z frame #4: <unknown function> + 0x208c3b (0x56305eb71c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0046292Z frame #5: <unknown function> + 0x14648d (0x56305eaaf48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0046868Z frame #6: PyObject_VectorcallMethod + 0x85 (0x56305eabe215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0047436Z frame #7: <unknown function> + 0x208ae7 (0x56305eb71ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0047985Z frame #8: <unknown function> + 0x208472 (0x56305eb71472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0048529Z frame #9: <unknown function> + 0x1da6ee (0x56305eb436ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0049087Z frame #10: <unknown function> + 0x1483bc (0x56305eab13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0049659Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0050238Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0050812Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x56305ea953bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0051379Z frame #14: <unknown function> + 0x147734 (0x56305eab0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0051949Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x56305ea97c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0052522Z frame #16: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0053082Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0053646Z frame #18: _PyFunction_Vectorcall + 0x6c (0x56305eaa50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0054210Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x56305ea957b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0054771Z frame #20: <unknown function> + 0x147734 (0x56305eab0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0055308Z frame #21: <unknown function> + 0x21ca82 (0x56305eb85a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0055843Z frame #22: <unknown function> + 0x21c9e4 (0x56305eb859e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0056439Z frame #23: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0056965Z frame #24: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0057282Z 
2025-07-17T11:07:26.0057379Z SIGABRT(6), PID: 1794251, Thread 1797828: 
2025-07-17T11:07:26.0058121Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0058889Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0059364Z frame #2: __poll + 0x4f (0x7f6044e86bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0060100Z frame #3: <unknown function> + 0x489f (0x7f604499289f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.0060827Z frame #4: <unknown function> + 0x1483bc (0x55d53e5a73bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0061393Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0062054Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0062643Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0063207Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0063773Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55d53e58fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0064415Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0064664Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d53e58dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0064898Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0065137Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0065368Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0065606Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0065837Z frame #16: <unknown function> + 0x147734 (0x55d53e5a6734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0066060Z frame #17: <unknown function> + 0x21ca82 (0x55d53e67ba82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0066286Z frame #18: <unknown function> + 0x21c9e4 (0x55d53e67b9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0066503Z frame #19: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0066725Z frame #20: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0066729Z 
2025-07-17T11:07:26.0066824Z SIGABRT(6), PID: 1794246, Thread 1799043: 
2025-07-17T11:07:26.0067324Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0067540Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0067760Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0067989Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fcf9bc012dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0068445Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fced6040048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0068871Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fced6042bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0069105Z frame #6: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0069323Z frame #7: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0069538Z frame #8: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0069820Z SIGABRT(6), PID: 1794250, Thread 1799145: 
2025-07-17T11:07:26.0070297Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0070514Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0070720Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0070992Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0071377Z frame #4: <unknown function> + 0x5a02023 (0x7fcbb8410023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0072010Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcbb84141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0072615Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcbb840b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0073065Z frame #7: <unknown function> + 0x79d655 (0x7fcbc98fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0073296Z frame #8: <unknown function> + 0xdc253 (0x7fcb80946253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0073516Z frame #9: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0073732Z frame #10: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0073736Z 
2025-07-17T11:07:26.0073740Z 
2025-07-17T11:07:26.0073839Z SIGABRT(6), PID: 1794249, Thread 1799123: 
2025-07-17T11:07:26.0074314Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0074529Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0074734Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0074943Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0075318Z frame #4: <unknown function> + 0x5a02023 (0x7f44c7101023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0075882Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f44c71051ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0076479Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f44c70fc1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0076882Z frame #7: <unknown function> + 0x79d655 (0x7f44d85ec655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0077115Z frame #8: <unknown function> + 0xdc253 (0x7f448f637253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0077332Z frame #9: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0077550Z frame #10: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0077561Z 
2025-07-17T11:07:26.0077654Z SIGABRT(6), PID: 1794246, Thread 1799044: 
2025-07-17T11:07:26.0078136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0078348Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0078638Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0078870Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fcf9bc012dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0079352Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fced603a741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0079638Z frame #5: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0079867Z frame #6: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0080081Z frame #7: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0080086Z 
2025-07-17T11:07:26.0080184Z SIGABRT(6), PID: 1794251, Thread 1797829: 
2025-07-17T11:07:26.0080714Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0080935Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0081145Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0081432Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6044e022dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0081872Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f5f7f240048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0082301Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f5f7f242bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0082529Z frame #6: <unknown function> + 0xdc253 (0x7f5f51764253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0082743Z frame #7: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0082961Z frame #8: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0082965Z 
2025-07-17T11:07:26.0083063Z SIGABRT(6), PID: 1794246, Thread 1799046: 
2025-07-17T11:07:26.0083537Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0083755Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0083925Z frame #2: ioctl + 0x3f (0x7fcf9bc8794f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0084169Z frame #3: <unknown function> + 0x13eef0 (0x7fce17f28ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0084402Z frame #4: <unknown function> + 0x1378c4 (0x7fce17f218c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0084638Z frame #5: <unknown function> + 0x95b2b (0x7fce17e7fb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0084868Z frame #6: <unknown function> + 0x34b71 (0x7fce17e1eb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0085085Z frame #7: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0085298Z frame #8: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0085394Z SIGABRT(6), PID: 1794251, Thread 1797830: 
2025-07-17T11:07:26.0085864Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0086076Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0086281Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0086512Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6044e022dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0087053Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f5f7f23a741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0087284Z frame #5: <unknown function> + 0xdc253 (0x7f5f51764253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0087496Z frame #6: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0087767Z frame #7: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0087772Z 
2025-07-17T11:07:26.0087775Z 
2025-07-17T11:07:26.0087867Z SIGABRT(6), PID: 1794247, Thread 1799112: 
2025-07-17T11:07:26.0088340Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0088604Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0088822Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0089025Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0089405Z frame #4: <unknown function> + 0x5a02023 (0x7f55b2297023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0090030Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f55b229b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0090633Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f55b22921d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0091019Z frame #7: <unknown function> + 0x79d655 (0x7f55c377e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0091250Z frame #8: <unknown function> + 0xdc253 (0x7f557a7cd253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0091467Z frame #9: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0091684Z frame #10: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0091688Z 
2025-07-17T11:07:26.0091790Z SIGABRT(6), PID: 1794251, Thread 1797848: 
2025-07-17T11:07:26.0092260Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0092473Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0092640Z frame #2: ioctl + 0x3f (0x7f6044e8894f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0092882Z frame #3: <unknown function> + 0x13eef0 (0x7f5ec1128ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0093120Z frame #4: <unknown function> + 0x1378c4 (0x7f5ec11218c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0093355Z frame #5: <unknown function> + 0x95b2b (0x7f5ec107fb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0093581Z frame #6: <unknown function> + 0x34b71 (0x7f5ec101eb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0093796Z frame #7: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0094008Z frame #8: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0094012Z 
2025-07-17T11:07:26.0094110Z SIGABRT(6), PID: 1794246, Thread 1799059: 
2025-07-17T11:07:26.0094579Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0094798Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0095040Z frame #2: __poll + 0x4f (0x7fcf9bc85bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0095259Z frame #3: <unknown function> + 0xd4540 (0x7fce60098540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0095471Z frame #4: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0095686Z frame #5: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0095691Z 
2025-07-17T11:07:26.0095861Z SIGABRT(6), PID: 1794249, Thread 1799124: 
2025-07-17T11:07:26.0096399Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0096607Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0096906Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0097115Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0097498Z frame #4: <unknown function> + 0x5a02023 (0x7f44c7101023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0098128Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f44c71051ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0098727Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f44c70fc1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0099110Z frame #7: <unknown function> + 0x79d655 (0x7f44d85ec655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0099345Z frame #8: <unknown function> + 0xdc253 (0x7f448f637253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0099557Z frame #9: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0099780Z frame #10: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0099877Z SIGABRT(6), PID: 1794246, Thread 1799064: 
2025-07-17T11:07:26.0100359Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0100571Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0100745Z frame #2: __poll + 0x4f (0x7fcf9bc85bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0100962Z frame #3: <unknown function> + 0xd65a2 (0x7fce6009a5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0101178Z frame #4: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0101394Z frame #5: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0101398Z 
2025-07-17T11:07:26.0101402Z 
2025-07-17T11:07:26.0101501Z SIGABRT(6), PID: 1794251, Thread 1799070: 
2025-07-17T11:07:26.0101973Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0102189Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0102350Z frame #2: __poll + 0x4f (0x7f6044e86bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0102569Z frame #3: <unknown function> + 0xd4540 (0x7f5f09298540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0102778Z frame #4: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0102993Z frame #5: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0103079Z 
2025-07-17T11:07:26.0103180Z SIGABRT(6), PID: 1794250, Thread 1799146: 
2025-07-17T11:07:26.0103650Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0103867Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0104078Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0104362Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0104741Z frame #4: <unknown function> + 0x5a02023 (0x7fcbb8410023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0105363Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcbb84141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0105962Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcbb840b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0106411Z frame #7: <unknown function> + 0x79d655 (0x7fcbc98fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0106645Z frame #8: <unknown function> + 0xdc253 (0x7fcb80946253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0106860Z frame #9: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0107078Z frame #10: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0107082Z 
2025-07-17T11:07:26.0107187Z SIGABRT(6), PID: 1794251, Thread 1799071: 
2025-07-17T11:07:26.0107658Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0107875Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0108037Z frame #2: __poll + 0x4f (0x7f6044e86bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0108261Z frame #3: <unknown function> + 0xd65a2 (0x7f5f0929a5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0108470Z frame #4: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0108684Z frame #5: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0108688Z 
2025-07-17T11:07:26.0108779Z SIGABRT(6), PID: 1794246, Thread 1799076: 
2025-07-17T11:07:26.0109255Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0109465Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0109681Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0109885Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0110107Z frame #4: <unknown function> + 0xd1e7f (0x7fce60095e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0110314Z frame #5: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0110529Z frame #6: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0110535Z 
2025-07-17T11:07:26.0110629Z SIGABRT(6), PID: 1794251, Thread 1799072: 
2025-07-17T11:07:26.0111104Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0111309Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0111593Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0111794Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0112014Z frame #4: <unknown function> + 0xd1e7f (0x7f5f09295e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0112219Z frame #5: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0112495Z frame #6: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0112499Z 
2025-07-17T11:07:26.0112593Z SIGABRT(6), PID: 1794246, Thread 1799084: 
2025-07-17T11:07:26.0113121Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0113338Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0113555Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0113764Z frame #3: <unknown function> + 0x9ca53 (0x7fcf9bc09a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0114085Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55a9caf26bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0114317Z frame #5: <unknown function> + 0x1d314f (0x55a9cafe514f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0114547Z frame #6: <unknown function> + 0x1d2f6b (0x55a9cafe4f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0114773Z frame #7: <unknown function> + 0x6df36 (0x55a9cae7ff36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0115022Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0115248Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0115499Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0115728Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0115972Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0116200Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0116440Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0116666Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0116914Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0117139Z frame #17: <unknown function> + 0x147734 (0x55a9caf59734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0117373Z frame #18: <unknown function> + 0x21ca82 (0x55a9cb02ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0117595Z frame #19: <unknown function> + 0x21c9e4 (0x55a9cb02e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0117817Z frame #20: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0118032Z frame #21: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0118036Z 
2025-07-17T11:07:26.0118137Z SIGABRT(6), PID: 1794251, Thread 1799083: 
2025-07-17T11:07:26.0118607Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0118825Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0119033Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0119418Z frame #3: <unknown function> + 0x9ca53 (0x7f6044e0aa53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0119668Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55d53e573bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0119902Z frame #5: <unknown function> + 0x1d314f (0x55d53e63214f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0120126Z frame #6: <unknown function> + 0x1d2f6b (0x55d53e631f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0120414Z frame #7: <unknown function> + 0x6df36 (0x55d53e4ccf36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0120664Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0120896Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0121191Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0121431Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0121666Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0121956Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0122191Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0122423Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0122658Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0122889Z frame #17: <unknown function> + 0x147734 (0x55d53e5a6734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0123112Z frame #18: <unknown function> + 0x21ca82 (0x55d53e67ba82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0123342Z frame #19: <unknown function> + 0x21c9e4 (0x55d53e67b9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0123553Z frame #20: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0123773Z frame #21: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0123780Z 
2025-07-17T11:07:26.0123872Z SIGABRT(6), PID: 1794246, Thread 1799099: 
2025-07-17T11:07:26.0124352Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0124573Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0124740Z frame #2: read + 0x4c (0x7fcf9bc8181c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0124934Z frame #3: _Py_read + 0x4c (0x55a9cafeca6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0125162Z frame #4: <unknown function> + 0x208c3b (0x55a9cb01ac3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0125392Z frame #5: <unknown function> + 0x14648d (0x55a9caf5848d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0125639Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55a9caf67215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0125873Z frame #7: <unknown function> + 0x208ae7 (0x55a9cb01aae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0126096Z frame #8: <unknown function> + 0x208472 (0x55a9cb01a472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0126323Z frame #9: <unknown function> + 0x1da6ee (0x55a9cafec6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0126548Z frame #10: <unknown function> + 0x1483bc (0x55a9caf5a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0126790Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0127023Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0127353Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55a9caf3e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0127582Z frame #14: <unknown function> + 0x147734 (0x55a9caf59734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0127833Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a9caf40c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0128120Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0128362Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0128593Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55a9caf4e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0128888Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55a9caf3e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0129119Z frame #20: <unknown function> + 0x147734 (0x55a9caf59734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0129343Z frame #21: <unknown function> + 0x21ca82 (0x55a9cb02ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0129569Z frame #22: <unknown function> + 0x21c9e4 (0x55a9cb02e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0129850Z frame #23: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0130070Z frame #24: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0130076Z 
2025-07-17T11:07:26.0130170Z SIGABRT(6), PID: 1794251, Thread 1799098: 
2025-07-17T11:07:26.0130644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0130862Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0131024Z frame #2: read + 0x4c (0x7f6044e8281c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0131213Z frame #3: _Py_read + 0x4c (0x55d53e639a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0131436Z frame #4: <unknown function> + 0x208c3b (0x55d53e667c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0131661Z frame #5: <unknown function> + 0x14648d (0x55d53e5a548d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0131905Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55d53e5b4215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0132133Z frame #7: <unknown function> + 0x208ae7 (0x55d53e667ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0132350Z frame #8: <unknown function> + 0x208472 (0x55d53e667472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0132573Z frame #9: <unknown function> + 0x1da6ee (0x55d53e6396ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0132800Z frame #10: <unknown function> + 0x1483bc (0x55d53e5a73bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0133045Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0133270Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0133513Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55d53e58b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0133738Z frame #14: <unknown function> + 0x147734 (0x55d53e5a6734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0133993Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d53e58dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0134222Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0134465Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0134692Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55d53e59b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0135001Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55d53e58b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0135221Z frame #20: <unknown function> + 0x147734 (0x55d53e5a6734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0135447Z frame #21: <unknown function> + 0x21ca82 (0x55d53e67ba82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0135668Z frame #22: <unknown function> + 0x21c9e4 (0x55d53e67b9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0136026Z frame #23: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0136258Z frame #24: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0136262Z 
2025-07-17T11:07:26.0136368Z SIGABRT(6), PID: 1794247, Thread 1799113: 
2025-07-17T11:07:26.0136921Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0137146Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0137353Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0137635Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0138013Z frame #4: <unknown function> + 0x5a02023 (0x7f55b2297023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0138582Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f55b229b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0139180Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f55b22921d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0139571Z frame #7: <unknown function> + 0x79d655 (0x7f55c377e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0139803Z frame #8: <unknown function> + 0xdc253 (0x7f557a7cd253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0140030Z frame #9: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0140248Z frame #10: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0140252Z 
2025-07-17T11:07:26.0140352Z SIGABRT(6), PID: 1794249, Thread 1799125: 
2025-07-17T11:07:26.0140833Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0141050Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0141257Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0141467Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0141839Z frame #4: <unknown function> + 0x5a02023 (0x7f44c7101023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0142407Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f44c71051ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0143000Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f44c70fc1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0143389Z frame #7: <unknown function> + 0x79d655 (0x7f44d85ec655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0143624Z frame #8: <unknown function> + 0xdc253 (0x7f448f637253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0152645Z frame #9: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0152922Z frame #10: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0152935Z 
2025-07-17T11:07:26.0153047Z SIGABRT(6), PID: 1794250, Thread 1799147: 
2025-07-17T11:07:26.0153724Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0153969Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0154184Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0154476Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0154881Z frame #4: <unknown function> + 0x5a02023 (0x7fcbb8410023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0155467Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcbb84141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0156162Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcbb840b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0156564Z frame #7: <unknown function> + 0x79d655 (0x7fcbc98fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0156816Z frame #8: <unknown function> + 0xdc253 (0x7fcb80946253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0157038Z frame #9: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0157265Z frame #10: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0157270Z 
2025-07-17T11:07:26.0157368Z SIGABRT(6), PID: 1794246, Thread 1799132: 
2025-07-17T11:07:26.0157858Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0158073Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0158284Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0158490Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0158873Z frame #4: <unknown function> + 0x5a02023 (0x7fcee002e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0159436Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcee00321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0160027Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcee00291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0160407Z frame #7: <unknown function> + 0x79d655 (0x7fcef1515655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0160644Z frame #8: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0160854Z frame #9: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0161073Z frame #10: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0161078Z 
2025-07-17T11:07:26.0161171Z SIGABRT(6), PID: 1794251, Thread 1799128: 
2025-07-17T11:07:26.0161720Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0161927Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0162133Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0162331Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0162766Z frame #4: <unknown function> + 0x5a02023 (0x7f5f8922e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0163393Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f892321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0163984Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f892291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0164367Z frame #7: <unknown function> + 0x79d655 (0x7f5f9a715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0164686Z frame #8: <unknown function> + 0xdc253 (0x7f5f51764253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0164901Z frame #9: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0165117Z frame #10: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0165122Z 
2025-07-17T11:07:26.0165216Z SIGABRT(6), PID: 1794247, Thread 1799114: 
2025-07-17T11:07:26.0165687Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0165898Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0166101Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0166302Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0166670Z frame #4: <unknown function> + 0x5a02023 (0x7f55b2297023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0167230Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f55b229b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0167816Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f55b22921d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0168196Z frame #7: <unknown function> + 0x79d655 (0x7f55c377e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0168425Z frame #8: <unknown function> + 0xdc253 (0x7f557a7cd253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0168641Z frame #9: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0168854Z frame #10: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0168859Z 
2025-07-17T11:07:26.0168955Z SIGABRT(6), PID: 1794249, Thread 1799126: 
2025-07-17T11:07:26.0169423Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0169635Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0169838Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0170114Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0170481Z frame #4: <unknown function> + 0x5a02023 (0x7f44c7101023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0171042Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f44c71051ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0171686Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f44c70fc1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0172129Z frame #7: <unknown function> + 0x79d655 (0x7f44d85ec655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0172360Z frame #8: <unknown function> + 0xdc253 (0x7f448f637253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0172574Z frame #9: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0172785Z frame #10: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0172848Z 
2025-07-17T11:07:26.0172944Z SIGABRT(6), PID: 1794250, Thread 1799148: 
2025-07-17T11:07:26.0173421Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0173626Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0173835Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0174035Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0174404Z frame #4: <unknown function> + 0x5a02023 (0x7fcbb8410023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0174964Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcbb84141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0175559Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcbb840b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0176012Z frame #7: <unknown function> + 0x79d655 (0x7fcbc98fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0176246Z frame #8: <unknown function> + 0xdc253 (0x7fcb80946253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0176459Z frame #9: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0176675Z frame #10: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0176680Z 
2025-07-17T11:07:26.0176771Z SIGABRT(6), PID: 1794246, Thread 1799133: 
2025-07-17T11:07:26.0177253Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0177466Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0177676Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0177874Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0178244Z frame #4: <unknown function> + 0x5a02023 (0x7fcee002e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0178797Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcee00321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0179489Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcee00291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0179873Z frame #7: <unknown function> + 0x79d655 (0x7fcef1515655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0180176Z frame #8: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0180393Z frame #9: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0180613Z frame #10: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0180687Z 
2025-07-17T11:07:26.0180782Z SIGABRT(6), PID: 1794251, Thread 1799129: 
2025-07-17T11:07:26.0181277Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0181488Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0181784Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0181983Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0182357Z frame #4: <unknown function> + 0x5a02023 (0x7f5f8922e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0182922Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f892321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0183519Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f892291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0183904Z frame #7: <unknown function> + 0x79d655 (0x7f5f9a715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0184133Z frame #8: <unknown function> + 0xdc253 (0x7f5f51764253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0184343Z frame #9: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0184559Z frame #10: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0184563Z 
2025-07-17T11:07:26.0184658Z SIGABRT(6), PID: 1794247, Thread 1799115: 
2025-07-17T11:07:26.0185131Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0185342Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0185543Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0185742Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0186112Z frame #4: <unknown function> + 0x5a02023 (0x7f55b2297023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0186675Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f55b229b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0187275Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f55b22921d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0187720Z frame #7: <unknown function> + 0x79d655 (0x7f55c377e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0187948Z frame #8: <unknown function> + 0xdc253 (0x7f557a7cd253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0188158Z frame #9: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0188373Z frame #10: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0188378Z 
2025-07-17T11:07:26.0188554Z SIGABRT(6), PID: 1794249, Thread 1799127: 
2025-07-17T11:07:26.0189025Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f44913cab9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0189292Z frame #1: <unknown function> + 0x42520 (0x7f4582c82520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0189498Z frame #2: <unknown function> + 0x91117 (0x7f4582cd1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0189702Z frame #3: pthread_cond_wait + 0x211 (0x7f4582cd3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0190067Z frame #4: <unknown function> + 0x5a02023 (0x7f44c7101023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0190686Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f44c71051ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0191274Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f44c70fc1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0191656Z frame #7: <unknown function> + 0x79d655 (0x7f44d85ec655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0191881Z frame #8: <unknown function> + 0xdc253 (0x7f448f637253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0192095Z frame #9: <unknown function> + 0x94ac3 (0x7f4582cd4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0192305Z frame #10: <unknown function> + 0x126850 (0x7f4582d66850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0192314Z 
2025-07-17T11:07:26.0192404Z SIGABRT(6), PID: 1794250, Thread 1799149: 
2025-07-17T11:07:26.0192882Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0193087Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0193295Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0193497Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0193867Z frame #4: <unknown function> + 0x5a02023 (0x7fcbb8410023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0194428Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcbb84141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0195028Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcbb840b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0195407Z frame #7: <unknown function> + 0x79d655 (0x7fcbc98fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0195633Z frame #8: <unknown function> + 0xdc253 (0x7fcb80946253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0195845Z frame #9: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0196135Z frame #10: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0196140Z 
2025-07-17T11:07:26.0196230Z SIGABRT(6), PID: 1794251, Thread 1799130: 
2025-07-17T11:07:26.0196704Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0196920Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0197185Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0197386Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0197758Z frame #4: <unknown function> + 0x5a02023 (0x7f5f8922e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0198395Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f892321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0199000Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f892291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0199437Z frame #7: <unknown function> + 0x79d655 (0x7f5f9a715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0199669Z frame #8: <unknown function> + 0xdc253 (0x7f5f51764253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0199877Z frame #9: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0200093Z frame #10: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0200099Z 
2025-07-17T11:07:26.0200189Z SIGABRT(6), PID: 1794246, Thread 1799135: 
2025-07-17T11:07:26.0200667Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0200876Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0201085Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0201283Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0201656Z frame #4: <unknown function> + 0x5a02023 (0x7fcee002e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0202215Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcee00321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0202802Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcee00291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0203191Z frame #7: <unknown function> + 0x79d655 (0x7fcef1515655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0203421Z frame #8: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0203633Z frame #9: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0203846Z frame #10: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0203850Z 
2025-07-17T11:07:26.0203942Z SIGABRT(6), PID: 1794247, Thread 1799116: 
2025-07-17T11:07:26.0204412Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0204694Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0204899Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0205098Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0205467Z frame #4: <unknown function> + 0x5a02023 (0x7f55b2297023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0206084Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f55b229b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0206728Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f55b22921d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0207111Z frame #7: <unknown function> + 0x79d655 (0x7f55c377e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0207337Z frame #8: <unknown function> + 0xdc253 (0x7f557a7cd253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0207612Z frame #9: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0207823Z frame #10: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0207827Z 
2025-07-17T11:07:26.0207926Z SIGABRT(6), PID: 1794250, Thread 1799150: 
2025-07-17T11:07:26.0208393Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0208605Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0208811Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0209013Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0209376Z frame #4: <unknown function> + 0x5a02023 (0x7fcbb8410023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0209940Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcbb84141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0210533Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcbb840b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0210909Z frame #7: <unknown function> + 0x79d655 (0x7fcbc98fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0211137Z frame #8: <unknown function> + 0xdc253 (0x7fcb80946253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0211345Z frame #9: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0211558Z frame #10: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0211563Z 
2025-07-17T11:07:26.0211653Z SIGABRT(6), PID: 1794251, Thread 1799131: 
2025-07-17T11:07:26.0212130Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0212338Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0212546Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0212741Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0213108Z frame #4: <unknown function> + 0x5a02023 (0x7f5f8922e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0213735Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f892321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0214384Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f892291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0214762Z frame #7: <unknown function> + 0x79d655 (0x7f5f9a715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0214990Z frame #8: <unknown function> + 0xdc253 (0x7f5f51764253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0215253Z frame #9: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0215471Z frame #10: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0215475Z 
2025-07-17T11:07:26.0215564Z SIGABRT(6), PID: 1794246, Thread 1799137: 
2025-07-17T11:07:26.0216116Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0216415Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0216629Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0216829Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0217200Z frame #4: <unknown function> + 0x5a02023 (0x7fcee002e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0217760Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcee00321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0218354Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcee00291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0218729Z frame #7: <unknown function> + 0x79d655 (0x7fcef1515655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0218959Z frame #8: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0219167Z frame #9: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0219381Z frame #10: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0219387Z 
2025-07-17T11:07:26.0219476Z SIGABRT(6), PID: 1794247, Thread 1799117: 
2025-07-17T11:07:26.0219945Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0220157Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0220360Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0220556Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0220919Z frame #4: <unknown function> + 0x5a02023 (0x7f55b2297023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0221480Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f55b229b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0222063Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f55b22921d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0222522Z frame #7: <unknown function> + 0x79d655 (0x7f55c377e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0222754Z frame #8: <unknown function> + 0xdc253 (0x7f557a7cd253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0223034Z frame #9: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0223249Z frame #10: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0223253Z 
2025-07-17T11:07:26.0223345Z SIGABRT(6), PID: 1794250, Thread 1799151: 
2025-07-17T11:07:26.0223884Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fcb826d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0224100Z frame #1: <unknown function> + 0x42520 (0x7fcc73f90520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0224304Z frame #2: <unknown function> + 0x91117 (0x7fcc73fdf117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0224506Z frame #3: pthread_cond_wait + 0x211 (0x7fcc73fe1a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0224935Z frame #4: <unknown function> + 0x5a02023 (0x7fcbb8410023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0225499Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcbb84141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0226086Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcbb840b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0226469Z frame #7: <unknown function> + 0x79d655 (0x7fcbc98fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0226693Z frame #8: <unknown function> + 0xdc253 (0x7fcb80946253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0226907Z frame #9: <unknown function> + 0x94ac3 (0x7fcc73fe2ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0227116Z frame #10: <unknown function> + 0x126850 (0x7fcc74074850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0227121Z 
2025-07-17T11:07:26.0227215Z SIGABRT(6), PID: 1794246, Thread 1799139: 
2025-07-17T11:07:26.0227693Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0227905Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0228110Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0228311Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0228679Z frame #4: <unknown function> + 0x5a02023 (0x7fcee002e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0229239Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcee00321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0229829Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcee00291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0230207Z frame #7: <unknown function> + 0x79d655 (0x7fcef1515655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0230434Z frame #8: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0230713Z frame #9: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0230930Z frame #10: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0230936Z 
2025-07-17T11:07:26.0231025Z SIGABRT(6), PID: 1794251, Thread 1799134: 
2025-07-17T11:07:26.0231578Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0231788Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0231992Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0232245Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0232620Z frame #4: <unknown function> + 0x5a02023 (0x7f5f8922e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0233177Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f892321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0233826Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f892291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0234204Z frame #7: <unknown function> + 0x79d655 (0x7f5f9a715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0234432Z frame #8: <unknown function> + 0xdc253 (0x7f5f51764253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0234640Z frame #9: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0234854Z frame #10: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0234861Z 
2025-07-17T11:07:26.0234951Z SIGABRT(6), PID: 1794247, Thread 1799118: 
2025-07-17T11:07:26.0235422Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0235629Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0235834Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0236029Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0236397Z frame #4: <unknown function> + 0x5a02023 (0x7f55b2297023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0236959Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f55b229b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0237553Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f55b22921d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0237934Z frame #7: <unknown function> + 0x79d655 (0x7f55c377e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0238162Z frame #8: <unknown function> + 0xdc253 (0x7f557a7cd253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0238370Z frame #9: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0238583Z frame #10: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0238588Z 
2025-07-17T11:07:26.0238683Z SIGABRT(6), PID: 1794246, Thread 1799140: 
2025-07-17T11:07:26.0239220Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0239429Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0239636Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0239835Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0240260Z frame #4: <unknown function> + 0x5a02023 (0x7fcee002e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0240894Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcee00321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0241481Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcee00291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0241864Z frame #7: <unknown function> + 0x79d655 (0x7fcef1515655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0242158Z frame #8: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0242373Z frame #9: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0242585Z frame #10: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0242588Z 
2025-07-17T11:07:26.0242680Z SIGABRT(6), PID: 1794247, Thread 1799119: 
2025-07-17T11:07:26.0243144Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f557c560b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0243357Z frame #1: <unknown function> + 0x42520 (0x7f566de1a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0243558Z frame #2: <unknown function> + 0x91117 (0x7f566de69117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0243756Z frame #3: pthread_cond_wait + 0x211 (0x7f566de6ba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0244121Z frame #4: <unknown function> + 0x5a02023 (0x7f55b2297023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0244678Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f55b229b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0245262Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f55b22921d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0245641Z frame #7: <unknown function> + 0x79d655 (0x7f55c377e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0245869Z frame #8: <unknown function> + 0xdc253 (0x7f557a7cd253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0246080Z frame #9: <unknown function> + 0x94ac3 (0x7f566de6cac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0246290Z frame #10: <unknown function> + 0x126850 (0x7f566defe850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0246296Z 
2025-07-17T11:07:26.0246386Z SIGABRT(6), PID: 1794251, Thread 1799136: 
2025-07-17T11:07:26.0246859Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0247065Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0247272Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0247538Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0247906Z frame #4: <unknown function> + 0x5a02023 (0x7f5f8922e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0248464Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f892321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0249107Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f892291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0249543Z frame #7: <unknown function> + 0x79d655 (0x7f5f9a715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0249774Z frame #8: <unknown function> + 0xdc253 (0x7f5f51764253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0249982Z frame #9: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0250194Z frame #10: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0250256Z 
2025-07-17T11:07:26.0250347Z SIGABRT(6), PID: 1794246, Thread 1799142: 
2025-07-17T11:07:26.0250825Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0251034Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0251240Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0251442Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0251814Z frame #4: <unknown function> + 0x5a02023 (0x7fcee002e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0252376Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcee00321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0252967Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcee00291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0253345Z frame #7: <unknown function> + 0x79d655 (0x7fcef1515655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0253575Z frame #8: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0253784Z frame #9: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0253999Z frame #10: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0254004Z 
2025-07-17T11:07:26.0254093Z SIGABRT(6), PID: 1794251, Thread 1799138: 
2025-07-17T11:07:26.0254563Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0254770Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0254976Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0255173Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0255539Z frame #4: <unknown function> + 0x5a02023 (0x7f5f8922e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0256167Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f892321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0256863Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f892291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0257251Z frame #7: <unknown function> + 0x79d655 (0x7f5f9a715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0257548Z frame #8: <unknown function> + 0xdc253 (0x7f5f51764253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0257760Z frame #9: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0257971Z frame #10: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0258048Z 
2025-07-17T11:07:26.0258144Z SIGABRT(6), PID: 1794246, Thread 1799143: 
2025-07-17T11:07:26.0258621Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fceaa2f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0258833Z frame #1: <unknown function> + 0x42520 (0x7fcf9bbaf520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0259214Z frame #2: <unknown function> + 0x91117 (0x7fcf9bbfe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0259416Z frame #3: pthread_cond_wait + 0x211 (0x7fcf9bc00a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0259787Z frame #4: <unknown function> + 0x5a02023 (0x7fcee002e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0260354Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fcee00321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0260941Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fcee00291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0261321Z frame #7: <unknown function> + 0x79d655 (0x7fcef1515655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0261550Z frame #8: <unknown function> + 0xdc253 (0x7fcea8564253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0261772Z frame #9: <unknown function> + 0x94ac3 (0x7fcf9bc01ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0261988Z frame #10: <unknown function> + 0x126850 (0x7fcf9bc93850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0261993Z 
2025-07-17T11:07:26.0262089Z SIGABRT(6), PID: 1794251, Thread 1799141: 
2025-07-17T11:07:26.0262561Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f534f7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0262780Z frame #1: <unknown function> + 0x42520 (0x7f6044db0520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0262984Z frame #2: <unknown function> + 0x91117 (0x7f6044dff117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0263184Z frame #3: pthread_cond_wait + 0x211 (0x7f6044e01a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0263558Z frame #4: <unknown function> + 0x5a02023 (0x7f5f8922e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0264125Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f892321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0264712Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f892291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0265092Z frame #7: <unknown function> + 0x79d655 (0x7f5f9a715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0265418Z frame #8: <unknown function> + 0xdc253 (0x7f5f51764253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0265630Z frame #9: <unknown function> + 0x94ac3 (0x7f6044e02ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0265845Z frame #10: <unknown function> + 0x126850 (0x7f6044e94850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0265848Z 
2025-07-17T11:07:26.0265996Z FAILED [25.1483s] [ 39%]
2025-07-17T11:07:26.0266002Z 
2025-07-17T11:07:26.0266118Z ==================================== RERUNS ====================================
2025-07-17T11:07:26.0266303Z __________________ TestMultiProc.test_ddp_optimizer_cudagraph __________________
2025-07-17T11:07:26.0266405Z Traceback (most recent call last):
2025-07-17T11:07:26.0266859Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in wrapper
2025-07-17T11:07:26.0266957Z     self._join_processes(fn)
2025-07-17T11:07:26.0267365Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 920, in _join_processes
2025-07-17T11:07:26.0267547Z     self._check_return_codes(fn, elapsed_time)
2025-07-17T11:07:26.0267968Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 996, in _check_return_codes
2025-07-17T11:07:26.0268050Z     self.assertEqual(
2025-07-17T11:07:26.0268418Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4176, in assertEqual
2025-07-17T11:07:26.0268571Z     raise error_metas.pop()[0].to_error(  # type: ignore[index]
2025-07-17T11:07:26.0268674Z AssertionError: Scalars are not equal!
2025-07-17T11:07:26.0268678Z 
2025-07-17T11:07:26.0268784Z Expected 0 but got -6.
2025-07-17T11:07:26.0268873Z Absolute difference: 6
2025-07-17T11:07:26.0268960Z Relative difference: inf
2025-07-17T11:07:26.0269086Z Expected exit code 0 but got -6 for pid: 1777265
2025-07-17T11:07:26.0269270Z __________________ TestMultiProc.test_ddp_optimizer_cudagraph __________________
2025-07-17T11:07:26.0269367Z Traceback (most recent call last):
2025-07-17T11:07:26.0269753Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in wrapper
2025-07-17T11:07:26.0269844Z     self._join_processes(fn)
2025-07-17T11:07:26.0270246Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 920, in _join_processes
2025-07-17T11:07:26.0270359Z     self._check_return_codes(fn, elapsed_time)
2025-07-17T11:07:26.0270764Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 996, in _check_return_codes
2025-07-17T11:07:26.0270853Z     self.assertEqual(
2025-07-17T11:07:26.0271214Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4176, in assertEqual
2025-07-17T11:07:26.0271372Z     raise error_metas.pop()[0].to_error(  # type: ignore[index]
2025-07-17T11:07:26.0271472Z AssertionError: Scalars are not equal!
2025-07-17T11:07:26.0271478Z 
2025-07-17T11:07:26.0271564Z Expected 0 but got -6.
2025-07-17T11:07:26.0271643Z Absolute difference: 6
2025-07-17T11:07:26.0271734Z Relative difference: inf
2025-07-17T11:07:26.0271853Z Expected exit code 0 but got -6 for pid: 1789276
2025-07-17T11:07:26.0271970Z =================================== FAILURES ===================================
2025-07-17T11:07:26.0272149Z __________________ TestMultiProc.test_ddp_optimizer_cudagraph __________________
2025-07-17T11:07:26.0272248Z Traceback (most recent call last):
2025-07-17T11:07:26.0272622Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in wrapper
2025-07-17T11:07:26.0272713Z     self._join_processes(fn)
2025-07-17T11:07:26.0273176Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 920, in _join_processes
2025-07-17T11:07:26.0273284Z     self._check_return_codes(fn, elapsed_time)
2025-07-17T11:07:26.0273699Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 996, in _check_return_codes
2025-07-17T11:07:26.0273784Z     self.assertEqual(
2025-07-17T11:07:26.0274200Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4176, in assertEqual
2025-07-17T11:07:26.0274351Z     raise error_metas.pop()[0].to_error(  # type: ignore[index]
2025-07-17T11:07:26.0274448Z AssertionError: Scalars are not equal!
2025-07-17T11:07:26.0274453Z 
2025-07-17T11:07:26.0274535Z Expected 0 but got -6.
2025-07-17T11:07:26.0274616Z Absolute difference: 6
2025-07-17T11:07:26.0274756Z Relative difference: inf
2025-07-17T11:07:26.0274879Z Expected exit code 0 but got -6 for pid: 1794246
2025-07-17T11:07:26.0275463Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-2c4842f0210cd3ab.xml -
2025-07-17T11:07:26.0275598Z =========================== short test summary info ============================
2025-07-17T11:07:26.0276073Z FAILED [25.1483s] distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph - AssertionError: Scalars are not equal!
2025-07-17T11:07:26.0276078Z 
2025-07-17T11:07:26.0276165Z Expected 0 but got -6.
2025-07-17T11:07:26.0276246Z Absolute difference: 6
2025-07-17T11:07:26.0276334Z Relative difference: inf
2025-07-17T11:07:26.0276447Z Expected exit code 0 but got -6 for pid: 1794246
2025-07-17T11:07:26.0276583Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-07-17T11:07:26.0276730Z =============== 1 failed, 8 passed, 2 rerun in 219.19s (0:03:39) ===============
2025-07-17T11:07:26.0277307Z /opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 24 leaked semaphore objects to clean up at shutdown
2025-07-17T11:07:26.0277454Z   warnings.warn('resource_tracker: There appear to be %d '
2025-07-17T11:07:26.0277535Z Got exit code 1
2025-07-17T11:07:26.0277614Z Retrying single test...
2025-07-17T11:07:26.0278637Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.0278719Z   import pkg_resources
2025-07-17T11:07:26.0279190Z Test results will be stored in test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-cf10226461b1eeac.xml
2025-07-17T11:07:26.0279308Z ============================= test session starts ==============================
2025-07-17T11:07:26.0279573Z platform linux -- Python 3.10.18, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python
2025-07-17T11:07:26.0279657Z cachedir: .pytest_cache
2025-07-17T11:07:26.0280028Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-07-17T11:07:26.0280124Z rootdir: /var/lib/jenkins/pytorch
2025-07-17T11:07:26.0280207Z configfile: pytest.ini
2025-07-17T11:07:26.0280579Z plugins: xdist-3.3.1, rerunfailures-14.0, xdoctest-1.1.0, flakefinder-1.1.0, cpp-2.3.0, subtests-0.13.1, hypothesis-5.35.1, typeguard-4.3.0
2025-07-17T11:07:26.0280745Z collecting ... collected 60 items / 22 deselected / 38 selected
2025-07-17T11:07:26.0281178Z stepcurrent: skipping 8 already run items. Running only test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph
2025-07-17T11:07:26.0281272Z Running 1 items in this shard
2025-07-17T11:07:26.0281344Z 
2025-07-17T11:07:26.0281995Z distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph I0717 11:01:10.691000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1800818
2025-07-17T11:07:26.0282358Z I0717 11:01:10.692000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1800819
2025-07-17T11:07:26.0282706Z I0717 11:01:10.693000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1800820
2025-07-17T11:07:26.0283110Z I0717 11:01:10.694000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1800821
2025-07-17T11:07:26.0283453Z I0717 11:01:10.695000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1800822
2025-07-17T11:07:26.0283857Z I0717 11:01:10.696000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1800823
2025-07-17T11:07:26.0284209Z I0717 11:01:10.697000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1800824
2025-07-17T11:07:26.0284558Z I0717 11:01:10.698000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1800825
2025-07-17T11:07:26.0285614Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.0285693Z   import pkg_resources
2025-07-17T11:07:26.0286677Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.0286759Z   import pkg_resources
2025-07-17T11:07:26.0287742Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.0287821Z   import pkg_resources
2025-07-17T11:07:26.0288798Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.0288877Z   import pkg_resources
2025-07-17T11:07:26.0289857Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.0289936Z   import pkg_resources
2025-07-17T11:07:26.0290912Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.0290993Z   import pkg_resources
2025-07-17T11:07:26.0291973Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.0292117Z   import pkg_resources
2025-07-17T11:07:26.0293093Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.0293176Z   import pkg_resources
2025-07-17T11:07:26.0293721Z [rank1]:W0717 11:01:32.480000 1800819 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.0294202Z [rank3]:W0717 11:01:32.480000 1800821 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.0294723Z [rank4]:W0717 11:01:32.481000 1800822 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.0295195Z [rank7]:W0717 11:01:32.481000 1800825 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.0295737Z [rank6]:W0717 11:01:32.481000 1800824 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.0296265Z [rank0]:W0717 11:01:32.481000 1800818 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.0296730Z [rank5]:W0717 11:01:32.481000 1800823 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.0297198Z [rank2]:W0717 11:01:32.483000 1800820 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.0297488Z Memory access fault by GPU node-9 (Agent handle: 0x561c1932eea0) on address 0x7fa04a403000. Reason: Unknown.
2025-07-17T11:07:26.0297769Z Memory access fault by GPU node-5 (Agent handle: 0x55c53f51fe40) on address 0x7fb8aa403000. Reason: Unknown.
2025-07-17T11:07:26.0298036Z Memory access fault by GPU node-3 (Agent handle: 0x555ecd2fa5b0) on address 0x7f9af3e03000. Reason: Unknown.
2025-07-17T11:07:26.0298307Z Memory access fault by GPU node-4 (Agent handle: 0x55727689dd50) on address 0x7f2236a1e000. Reason: Unknown.
2025-07-17T11:07:26.0298573Z Memory access fault by GPU node-8 (Agent handle: 0x55903f667ae0) on address 0x7f334be0a000. Reason: Unknown.
2025-07-17T11:07:26.0298854Z Memory access fault by GPU node-7 (Agent handle: 0x556dae0c4a40) on address 0x7f30d3403000. Reason: Unknown.
2025-07-17T11:07:26.0299118Z Memory access fault by GPU node-6 (Agent handle: 0x562c810dd510) on address 0x7f1919c03000. Reason: Unknown.
2025-07-17T11:07:26.0299392Z Memory access fault by GPU node-2 (Agent handle: 0x5644b98d5c30) on address 0x7f98af41d000. Reason: Unknown.
2025-07-17T11:07:26.0299491Z SIGABRT(6), PID: 1800821, Thread 1800821: 
2025-07-17T11:07:26.0299993Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0300219Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0300470Z frame #2: <unknown function> + 0x77426 (0x7fc0e17fe426 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0300705Z frame #3: <unknown function> + 0x76c8e (0x7fc0e17fdc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0300939Z frame #4: <unknown function> + 0x6ad69 (0x7fc0e17f1d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0301155Z frame #5: <unknown function> + 0x3f54db (0x7fc1724564db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0301479Z frame #6: <unknown function> + 0x3e11d8 (0x7fc1724421d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0301696Z frame #7: <unknown function> + 0xb2f86 (0x7fc172113f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0301914Z frame #8: <unknown function> + 0x8e501 (0x7fc1720ef501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0302383Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fc19cef67ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.0302787Z frame #10: <unknown function> + 0xd307d9 (0x7fc1bb4497d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0303026Z frame #11: <unknown function> + 0x139902 (0x55c5354b6902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0303368Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55c5354ada65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0303613Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0303875Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55c5354ada65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0304109Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0304435Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0304662Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0304905Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0305129Z frame #19: <unknown function> + 0x1474b2 (0x55c5354c44b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0305354Z frame #20: PyObject_Call + 0xbc (0x55c5354c4e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0305586Z frame #21: <unknown function> + 0x1ff69d (0x55c53557c69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0305820Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55c5354b2023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0306069Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55c5354adcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0306303Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0306543Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55c5354a93bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0306773Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0307010Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55c5354a93bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0307247Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0307491Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55c5354ada65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0307756Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55c5354b14b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0307975Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55c5354c2ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0308202Z frame #32: <unknown function> + 0x205949 (0x55c535582949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0308427Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55c5354b2023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0308671Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55c5354adcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0308896Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0309137Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55c5354a93bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0309361Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0309670Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55c5354a93bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0309892Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0310133Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55c5354a93bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0310358Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0310665Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55c5354aa3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0310896Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0311363Z frame #44: <unknown function> + 0x7ab3f7 (0x7fc1baec43f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0311589Z frame #45: <unknown function> + 0x13bc88 (0x55c5354b8c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0311803Z frame #46: PyObject_Call + 0x20f (0x55c5354c4f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0312046Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55c5354ae98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0312328Z frame #48: <unknown function> + 0x147608 (0x55c5354c4608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0312571Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c5354abc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0312803Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0313036Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55c5354a93bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0313269Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0313506Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55c5354aa3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0313732Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0313972Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55c5354a93bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0314200Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0314439Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c5354abc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0314670Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0314907Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c5354abc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0315132Z frame #60: <unknown function> + 0x147608 (0x55c5354c4608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0315372Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c5354abc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0315597Z frame #62: <unknown function> + 0x147608 (0x55c5354c4608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0315602Z 
2025-07-17T11:07:26.0315697Z SIGABRT(6), PID: 1800821, Thread 1800913: 
2025-07-17T11:07:26.0316187Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0316407Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0316617Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0316821Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0317338Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0317633Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0317860Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0317866Z 
2025-07-17T11:07:26.0317961Z SIGABRT(6), PID: 1800821, Thread 1800917: 
2025-07-17T11:07:26.0318458Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0318736Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0318951Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0319162Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0319729Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0319954Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0320173Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0320237Z 
2025-07-17T11:07:26.0320338Z SIGABRT(6), PID: 1800821, Thread 1800921: 
2025-07-17T11:07:26.0320831Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0321046Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0321249Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0321454Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0321961Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0322177Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0322385Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0322390Z 
2025-07-17T11:07:26.0322488Z SIGABRT(6), PID: 1800821, Thread 1800925: 
2025-07-17T11:07:26.0322965Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0323176Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0323381Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0323583Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0324085Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0324299Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0324508Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0324512Z 
2025-07-17T11:07:26.0324610Z SIGABRT(6), PID: 1800821, Thread 1800928: 
2025-07-17T11:07:26.0325080Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0325290Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0325497Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0325769Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0326265Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0326483Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0326767Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0326772Z 
2025-07-17T11:07:26.0326874Z SIGABRT(6), PID: 1800821, Thread 1800931: 
2025-07-17T11:07:26.0327357Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0327652Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0327862Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0328066Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0328560Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0328842Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0329056Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0329065Z 
2025-07-17T11:07:26.0329160Z SIGABRT(6), PID: 1800821, Thread 1800934: 
2025-07-17T11:07:26.0329640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0329849Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0330062Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0330258Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0330763Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0330977Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0331190Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0331194Z 
2025-07-17T11:07:26.0331286Z SIGABRT(6), PID: 1800821, Thread 1800937: 
2025-07-17T11:07:26.0331761Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0331970Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0332179Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0332377Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0332882Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0333093Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0333311Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0333316Z 
2025-07-17T11:07:26.0333408Z SIGABRT(6), PID: 1800821, Thread 1800940: 
2025-07-17T11:07:26.0333887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0334164Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0334373Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0334572Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0335125Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0335338Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0335552Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0335557Z 
2025-07-17T11:07:26.0335706Z SIGABRT(6), PID: 1800825, Thread 1800825: 
2025-07-17T11:07:26.0336265Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0336479Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0336809Z frame #2: <unknown function> + 0x7741f (0x7fa88192741f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0337047Z frame #3: <unknown function> + 0x76c8e (0x7fa881926c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0337282Z frame #4: <unknown function> + 0x6ad69 (0x7fa88191ad69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0337501Z frame #5: <unknown function> + 0x3f54db (0x7fa91257f4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0337718Z frame #6: <unknown function> + 0x3e11d8 (0x7fa91256b1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0337928Z frame #7: <unknown function> + 0xb2f86 (0x7fa91223cf86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0338144Z frame #8: <unknown function> + 0x8e501 (0x7fa912218501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0338543Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fa93d01f7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.0338942Z frame #10: <unknown function> + 0xd307d9 (0x7fa95b5727d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0339175Z frame #11: <unknown function> + 0x139902 (0x561c0fb27902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0339437Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x561c0fb1ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0339674Z frame #13: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0339926Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x561c0fb1ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0340153Z frame #15: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0340401Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0340625Z frame #17: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0340872Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0341093Z frame #19: <unknown function> + 0x1474b2 (0x561c0fb354b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0341305Z frame #20: PyObject_Call + 0xbc (0x561c0fb35e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0341529Z frame #21: <unknown function> + 0x1ff69d (0x561c0fbed69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0341762Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x561c0fb23023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0342007Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x561c0fb1ecc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0342328Z frame #24: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0342575Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x561c0fb1a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0342814Z frame #26: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0343059Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x561c0fb1a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0343364Z frame #28: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0343616Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x561c0fb1ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0343881Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x561c0fb224b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0344176Z frame #31: _PyObject_Call_Prepend + 0x69 (0x561c0fb33ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0344406Z frame #32: <unknown function> + 0x205949 (0x561c0fbf3949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0344632Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x561c0fb23023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0344940Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x561c0fb1ecc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0345166Z frame #35: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0345408Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x561c0fb1a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0345631Z frame #37: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0345869Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x561c0fb1a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0346095Z frame #39: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0346334Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x561c0fb1a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0346562Z frame #41: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0346810Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x561c0fb1b3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0347034Z frame #43: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0347437Z frame #44: <unknown function> + 0x7ab3f7 (0x7fa95afed3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0347658Z frame #45: <unknown function> + 0x13bc88 (0x561c0fb29c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0347869Z frame #46: PyObject_Call + 0x20f (0x561c0fb35f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0348118Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x561c0fb1f98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0348345Z frame #48: <unknown function> + 0x147608 (0x561c0fb35608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0348587Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x561c0fb1cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0348821Z frame #50: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0349055Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x561c0fb1a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0349283Z frame #52: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0349521Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x561c0fb1b3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0349752Z frame #54: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0349991Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x561c0fb1a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0350290Z frame #56: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0350531Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x561c0fb1cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0350760Z frame #58: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0350999Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x561c0fb1cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0351285Z frame #60: <unknown function> + 0x147608 (0x561c0fb35608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0351531Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x561c0fb1cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0351762Z frame #62: <unknown function> + 0x147608 (0x561c0fb35608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0351824Z 
2025-07-17T11:07:26.0351923Z SIGABRT(6), PID: 1800821, Thread 1800943: 
2025-07-17T11:07:26.0352414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0352635Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0352911Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0353117Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0353634Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0353842Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0354059Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0354063Z 
2025-07-17T11:07:26.0354161Z SIGABRT(6), PID: 1800825, Thread 1801164: 
2025-07-17T11:07:26.0354646Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0354856Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0355075Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0355279Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0355794Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0356015Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0356227Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0356233Z 
2025-07-17T11:07:26.0356338Z SIGABRT(6), PID: 1800821, Thread 1800946: 
2025-07-17T11:07:26.0356811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0357031Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0357239Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0357443Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0357941Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0358161Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0358445Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0358546Z SIGABRT(6), PID: 1800825, Thread 1801168: 
2025-07-17T11:07:26.0359017Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0359234Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0359520Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0359734Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0360292Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0360515Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0360729Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0360735Z 
2025-07-17T11:07:26.0360739Z 
2025-07-17T11:07:26.0360836Z SIGABRT(6), PID: 1800821, Thread 1800949: 
2025-07-17T11:07:26.0361366Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0361586Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0361793Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0361997Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0362496Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0362712Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0362921Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0362926Z 
2025-07-17T11:07:26.0363029Z SIGABRT(6), PID: 1800825, Thread 1801172: 
2025-07-17T11:07:26.0363501Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0363719Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0363923Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0364128Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0364623Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0364837Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0365044Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0365050Z 
2025-07-17T11:07:26.0365152Z SIGABRT(6), PID: 1800821, Thread 1800952: 
2025-07-17T11:07:26.0365628Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0365846Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0366055Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0366256Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0366825Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0367039Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0367273Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0367277Z 
2025-07-17T11:07:26.0367371Z SIGABRT(6), PID: 1800825, Thread 1801176: 
2025-07-17T11:07:26.0367912Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0368126Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0368397Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0368596Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0369104Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0369378Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0369595Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0369690Z SIGABRT(6), PID: 1800821, Thread 1800955: 
2025-07-17T11:07:26.0370171Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0370379Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0370588Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0370785Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0371283Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0371494Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0371709Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0371715Z 
2025-07-17T11:07:26.0371718Z 
2025-07-17T11:07:26.0371808Z SIGABRT(6), PID: 1800825, Thread 1801180: 
2025-07-17T11:07:26.0372286Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0372496Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0372707Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0372906Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0373415Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0373626Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0373847Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0373851Z 
2025-07-17T11:07:26.0373946Z SIGABRT(6), PID: 1800821, Thread 1800958: 
2025-07-17T11:07:26.0374431Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0374639Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0374932Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0375131Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0375639Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0375974Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0376211Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0376217Z 
2025-07-17T11:07:26.0376311Z SIGABRT(6), PID: 1800821, Thread 1800961: 
2025-07-17T11:07:26.0376878Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0377110Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0377321Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0377623Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0378141Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0378360Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0378572Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0378676Z SIGABRT(6), PID: 1800825, Thread 1801184: 
2025-07-17T11:07:26.0379162Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0379382Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0379591Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0379804Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0380312Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0380535Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0380752Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0380756Z 
2025-07-17T11:07:26.0380759Z 
2025-07-17T11:07:26.0380862Z SIGABRT(6), PID: 1800825, Thread 1801188: 
2025-07-17T11:07:26.0381333Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0381558Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0381766Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0381978Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0382483Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0382710Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0382921Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0382926Z 
2025-07-17T11:07:26.0383114Z SIGABRT(6), PID: 1800821, Thread 1800964: 
2025-07-17T11:07:26.0383592Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0383810Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0384020Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0384299Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0384800Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0385022Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0385290Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0385297Z 
2025-07-17T11:07:26.0385403Z SIGABRT(6), PID: 1800821, Thread 1800967: 
2025-07-17T11:07:26.0385878Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0386156Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0386360Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0386566Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0387065Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0387290Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0387505Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0387518Z 
2025-07-17T11:07:26.0387610Z SIGABRT(6), PID: 1800825, Thread 1801192: 
2025-07-17T11:07:26.0388097Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0388309Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0388528Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0388728Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0389231Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0389443Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0389663Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0389667Z 
2025-07-17T11:07:26.0389760Z SIGABRT(6), PID: 1800821, Thread 1800970: 
2025-07-17T11:07:26.0390238Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0390450Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0390663Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0390865Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0391370Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0391670Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0391888Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0391894Z 
2025-07-17T11:07:26.0391989Z SIGABRT(6), PID: 1800825, Thread 1801196: 
2025-07-17T11:07:26.0392475Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0392740Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0392959Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0393160Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0393835Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0394052Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0394274Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0394334Z 
2025-07-17T11:07:26.0394430Z SIGABRT(6), PID: 1800821, Thread 1800973: 
2025-07-17T11:07:26.0394919Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0395128Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0395339Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0395541Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0396048Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0396260Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0396478Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0396482Z 
2025-07-17T11:07:26.0396575Z SIGABRT(6), PID: 1800821, Thread 1800976: 
2025-07-17T11:07:26.0397056Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0397269Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0397490Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0397687Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0398195Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0398406Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0398625Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0398629Z 
2025-07-17T11:07:26.0398724Z SIGABRT(6), PID: 1800825, Thread 1801200: 
2025-07-17T11:07:26.0399211Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0399427Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0399634Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0399914Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0400412Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0400632Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0400842Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0401002Z SIGABRT(6), PID: 1800821, Thread 1800979: 
2025-07-17T11:07:26.0401481Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0401760Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0401972Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0402188Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0402688Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0402972Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0403188Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0403192Z 
2025-07-17T11:07:26.0403195Z 
2025-07-17T11:07:26.0403300Z SIGABRT(6), PID: 1800821, Thread 1800982: 
2025-07-17T11:07:26.0403773Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0403992Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0404203Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0404415Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0404913Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0405138Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0405346Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0405349Z 
2025-07-17T11:07:26.0405447Z SIGABRT(6), PID: 1800825, Thread 1801204: 
2025-07-17T11:07:26.0405925Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0406144Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0406352Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0406559Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0407056Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0407274Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0407487Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0407491Z 
2025-07-17T11:07:26.0407592Z SIGABRT(6), PID: 1800821, Thread 1800985: 
2025-07-17T11:07:26.0408062Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0408370Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0408578Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0408788Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0409345Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0409565Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0409775Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0409785Z 
2025-07-17T11:07:26.0409934Z SIGABRT(6), PID: 1800821, Thread 1800988: 
2025-07-17T11:07:26.0410425Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0410637Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0410914Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0411116Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0411618Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0411828Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0412047Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0412140Z SIGABRT(6), PID: 1800825, Thread 1801208: 
2025-07-17T11:07:26.0412626Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0412834Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0413048Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0413248Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0413752Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0413962Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0414187Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0414192Z 
2025-07-17T11:07:26.0414197Z 
2025-07-17T11:07:26.0414290Z SIGABRT(6), PID: 1800821, Thread 1800992: 
2025-07-17T11:07:26.0414770Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0414979Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0415193Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0415398Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0415900Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0416176Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0416391Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0416566Z SIGABRT(6), PID: 1800825, Thread 1801211: 
2025-07-17T11:07:26.0417051Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0417260Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0417543Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0417746Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0418253Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0418540Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0418767Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0418771Z 
2025-07-17T11:07:26.0418775Z 
2025-07-17T11:07:26.0418871Z SIGABRT(6), PID: 1800825, Thread 1801215: 
2025-07-17T11:07:26.0419450Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0419660Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0419876Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0420073Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0420579Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0420790Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0421014Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0421018Z 
2025-07-17T11:07:26.0421120Z SIGABRT(6), PID: 1800821, Thread 1800996: 
2025-07-17T11:07:26.0421593Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0421814Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0422022Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0422228Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0422725Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0422945Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0423153Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0423159Z 
2025-07-17T11:07:26.0423262Z SIGABRT(6), PID: 1800825, Thread 1801219: 
2025-07-17T11:07:26.0423735Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0423951Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0424156Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0424364Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0424860Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0425148Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0425364Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0425368Z 
2025-07-17T11:07:26.0425473Z SIGABRT(6), PID: 1800821, Thread 1801001: 
2025-07-17T11:07:26.0426005Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0426224Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0426433Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0426698Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0427206Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0427428Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0427700Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0427704Z 
2025-07-17T11:07:26.0427806Z SIGABRT(6), PID: 1800825, Thread 1801223: 
2025-07-17T11:07:26.0428287Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0428506Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0428719Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0428932Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0429430Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0429652Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0429863Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0429868Z 
2025-07-17T11:07:26.0429971Z SIGABRT(6), PID: 1800821, Thread 1801005: 
2025-07-17T11:07:26.0430446Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0430662Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0430865Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0431071Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0431562Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0431781Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0431999Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0432002Z 
2025-07-17T11:07:26.0432094Z SIGABRT(6), PID: 1800825, Thread 1801227: 
2025-07-17T11:07:26.0432573Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0432782Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0433088Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0433290Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0433799Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0434066Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0434288Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0434291Z 
2025-07-17T11:07:26.0434385Z SIGABRT(6), PID: 1800821, Thread 1801008: 
2025-07-17T11:07:26.0434929Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0435141Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0435360Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0435561Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0436127Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0436340Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0436558Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0436562Z 
2025-07-17T11:07:26.0436654Z SIGABRT(6), PID: 1800825, Thread 1801231: 
2025-07-17T11:07:26.0437142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0437355Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0437571Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0437779Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0438283Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0438493Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0438710Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0438714Z 
2025-07-17T11:07:26.0438809Z SIGABRT(6), PID: 1800821, Thread 1801012: 
2025-07-17T11:07:26.0439290Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0439501Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0439714Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0439914Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0440419Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0440628Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0440847Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0440852Z 
2025-07-17T11:07:26.0440946Z SIGABRT(6), PID: 1800825, Thread 1801235: 
2025-07-17T11:07:26.0441500Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0441706Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0441926Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0442122Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0442683Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0442905Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0443173Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0443180Z 
2025-07-17T11:07:26.0443284Z SIGABRT(6), PID: 1800821, Thread 1801017: 
2025-07-17T11:07:26.0443759Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0444033Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0444238Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0444447Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0444944Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0445159Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0445369Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0445375Z 
2025-07-17T11:07:26.0445475Z SIGABRT(6), PID: 1800825, Thread 1801238: 
2025-07-17T11:07:26.0445950Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0446168Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0446376Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0446589Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0447089Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0447311Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0447521Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0447625Z SIGABRT(6), PID: 1800821, Thread 1801021: 
2025-07-17T11:07:26.0448099Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0448314Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0448534Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0448747Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0449247Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0449466Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0449751Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0449755Z 
2025-07-17T11:07:26.0449758Z 
2025-07-17T11:07:26.0449868Z SIGABRT(6), PID: 1800825, Thread 1801241: 
2025-07-17T11:07:26.0450356Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0450645Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0450859Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0451077Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0451665Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0451894Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0452106Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0452166Z 
2025-07-17T11:07:26.0452272Z SIGABRT(6), PID: 1800821, Thread 1801025: 
2025-07-17T11:07:26.0452753Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0452975Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0453187Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0453389Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0453899Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0454114Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0454333Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0454338Z 
2025-07-17T11:07:26.0454433Z SIGABRT(6), PID: 1800825, Thread 1801244: 
2025-07-17T11:07:26.0454917Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0455128Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0455339Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0455542Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0456096Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0456308Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0456528Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0456532Z 
2025-07-17T11:07:26.0456625Z SIGABRT(6), PID: 1800821, Thread 1801029: 
2025-07-17T11:07:26.0457106Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0457314Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0457528Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0457729Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0458323Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0458539Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0458755Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0458759Z 
2025-07-17T11:07:26.0458923Z SIGABRT(6), PID: 1800825, Thread 1801247: 
2025-07-17T11:07:26.0459414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0459622Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0459910Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0460129Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0460647Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0460931Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0461158Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0461162Z 
2025-07-17T11:07:26.0461258Z SIGABRT(6), PID: 1800821, Thread 1801033: 
2025-07-17T11:07:26.0461747Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0461960Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0462182Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0462383Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0462913Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0463133Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0463354Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0463358Z 
2025-07-17T11:07:26.0463451Z SIGABRT(6), PID: 1800825, Thread 1801250: 
2025-07-17T11:07:26.0463936Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0464149Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0464359Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0464566Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0465068Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0465294Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0465501Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0465507Z 
2025-07-17T11:07:26.0465613Z SIGABRT(6), PID: 1800821, Thread 1801037: 
2025-07-17T11:07:26.0466091Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0466387Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0466600Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0466813Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0467375Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0467609Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0467820Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0467824Z 
2025-07-17T11:07:26.0467923Z SIGABRT(6), PID: 1800825, Thread 1801253: 
2025-07-17T11:07:26.0468453Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0468682Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0468896Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0469173Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0469677Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0469901Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0470118Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0470121Z 
2025-07-17T11:07:26.0470228Z SIGABRT(6), PID: 1800821, Thread 1801041: 
2025-07-17T11:07:26.0470706Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0470931Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0471144Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0471356Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0471859Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0472080Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0472291Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0472295Z 
2025-07-17T11:07:26.0472400Z SIGABRT(6), PID: 1800825, Thread 1801256: 
2025-07-17T11:07:26.0472870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0473093Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0473302Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0473522Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0474026Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0474258Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0474474Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0474557Z 
2025-07-17T11:07:26.0474656Z SIGABRT(6), PID: 1800821, Thread 1801044: 
2025-07-17T11:07:26.0475136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0475347Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0475567Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0475830Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0476343Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0476615Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0476845Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0476849Z 
2025-07-17T11:07:26.0476945Z SIGABRT(6), PID: 1800825, Thread 1801259: 
2025-07-17T11:07:26.0477435Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0477711Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0477927Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0478129Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0478637Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0478844Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0479071Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0479075Z 
2025-07-17T11:07:26.0479173Z SIGABRT(6), PID: 1800821, Thread 1801049: 
2025-07-17T11:07:26.0479657Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0479864Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0480078Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0480275Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0480775Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0480986Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0481200Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0481206Z 
2025-07-17T11:07:26.0481300Z SIGABRT(6), PID: 1800825, Thread 1801261: 
2025-07-17T11:07:26.0481779Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0481988Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0482201Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0482401Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0482903Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0483184Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0483399Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0483405Z 
2025-07-17T11:07:26.0483496Z SIGABRT(6), PID: 1800821, Thread 1801053: 
2025-07-17T11:07:26.0484053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0484264Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0484474Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0484728Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0485233Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0485443Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0485735Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0485739Z 
2025-07-17T11:07:26.0485832Z SIGABRT(6), PID: 1800825, Thread 1801264: 
2025-07-17T11:07:26.0486312Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0486524Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0486732Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0486940Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0487442Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0487661Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0487873Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0487876Z 
2025-07-17T11:07:26.0487977Z SIGABRT(6), PID: 1800821, Thread 1801057: 
2025-07-17T11:07:26.0488451Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0488660Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0488869Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0489075Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0489574Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0489791Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0490001Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0490005Z 
2025-07-17T11:07:26.0490105Z SIGABRT(6), PID: 1800825, Thread 1801268: 
2025-07-17T11:07:26.0490574Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0490791Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0490994Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0491272Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0491770Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0491990Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0492264Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0492269Z 
2025-07-17T11:07:26.0492371Z SIGABRT(6), PID: 1800821, Thread 1801061: 
2025-07-17T11:07:26.0492900Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0493120Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0493329Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0493536Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0494097Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0494312Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0494519Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0494523Z 
2025-07-17T11:07:26.0494622Z SIGABRT(6), PID: 1800825, Thread 1801270: 
2025-07-17T11:07:26.0495097Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0495316Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0495523Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0495731Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0496295Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0496512Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0496723Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0496734Z 
2025-07-17T11:07:26.0496827Z SIGABRT(6), PID: 1800825, Thread 1801273: 
2025-07-17T11:07:26.0497312Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0497524Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0497739Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0497941Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0498449Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0498662Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0498880Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0498883Z 
2025-07-17T11:07:26.0498976Z SIGABRT(6), PID: 1800821, Thread 1801065: 
2025-07-17T11:07:26.0499455Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0499747Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0499966Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0500164Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0500749Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0500961Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0501248Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0501255Z 
2025-07-17T11:07:26.0501349Z SIGABRT(6), PID: 1800821, Thread 1801069: 
2025-07-17T11:07:26.0501832Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0502041Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0502354Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0502568Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0503088Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0503307Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0503525Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0503531Z 
2025-07-17T11:07:26.0503625Z SIGABRT(6), PID: 1800825, Thread 1801276: 
2025-07-17T11:07:26.0504116Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0504328Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0504541Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0504743Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0505251Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0505462Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0505680Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0505686Z 
2025-07-17T11:07:26.0505779Z SIGABRT(6), PID: 1800821, Thread 1801075: 
2025-07-17T11:07:26.0506261Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0506471Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0506688Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0506885Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0507392Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0507603Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0507891Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0507895Z 
2025-07-17T11:07:26.0507993Z SIGABRT(6), PID: 1800825, Thread 1801278: 
2025-07-17T11:07:26.0508473Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0508743Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0508953Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0509161Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0509718Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0509939Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0510147Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0510151Z 
2025-07-17T11:07:26.0510310Z SIGABRT(6), PID: 1800821, Thread 1801080: 
2025-07-17T11:07:26.0510783Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0510999Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0511204Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0511409Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0511905Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0512126Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0512334Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0512339Z 
2025-07-17T11:07:26.0512441Z SIGABRT(6), PID: 1800825, Thread 1801280: 
2025-07-17T11:07:26.0512920Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0513141Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0513348Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0513558Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0514061Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0514281Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0514492Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0514496Z 
2025-07-17T11:07:26.0514597Z SIGABRT(6), PID: 1800821, Thread 1801084: 
2025-07-17T11:07:26.0515070Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0515285Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0515491Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0515700Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0516297Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0516514Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0516724Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0516728Z 
2025-07-17T11:07:26.0516891Z SIGABRT(6), PID: 1800825, Thread 1801282: 
2025-07-17T11:07:26.0517367Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0517580Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0517843Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0518056Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0518558Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0518841Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0519054Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0519066Z 
2025-07-17T11:07:26.0519162Z SIGABRT(6), PID: 1800821, Thread 1801088: 
2025-07-17T11:07:26.0519649Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0519859Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0520079Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0520281Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0520786Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0520999Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0521222Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0521226Z 
2025-07-17T11:07:26.0521317Z SIGABRT(6), PID: 1800825, Thread 1801284: 
2025-07-17T11:07:26.0521799Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0522011Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0522227Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0522427Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0522933Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0523145Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0523360Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0523364Z 
2025-07-17T11:07:26.0523459Z SIGABRT(6), PID: 1800821, Thread 1801092: 
2025-07-17T11:07:26.0523946Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0524223Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0524434Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0524632Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0525137Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0525505Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0525724Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0525728Z 
2025-07-17T11:07:26.0525821Z SIGABRT(6), PID: 1800825, Thread 1801286: 
2025-07-17T11:07:26.0526362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0526575Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0526785Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0527043Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0527549Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0527758Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0527974Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0527978Z 
2025-07-17T11:07:26.0528072Z SIGABRT(6), PID: 1800821, Thread 1801097: 
2025-07-17T11:07:26.0528553Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0528761Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0528974Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0529170Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0529672Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0529885Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0530091Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0530097Z 
2025-07-17T11:07:26.0530196Z SIGABRT(6), PID: 1800821, Thread 1801101: 
2025-07-17T11:07:26.0530669Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0530884Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0531089Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0531292Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0531783Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0532001Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0532215Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0532381Z SIGABRT(6), PID: 1800825, Thread 1801288: 
2025-07-17T11:07:26.0532854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0533069Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0533277Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0533538Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0534037Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0534253Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0534524Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0534530Z 
2025-07-17T11:07:26.0534534Z 
2025-07-17T11:07:26.0534632Z SIGABRT(6), PID: 1800821, Thread 1801105: 
2025-07-17T11:07:26.0535107Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0535376Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0535586Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0535793Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0536416Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0536633Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0536846Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0536947Z SIGABRT(6), PID: 1800825, Thread 1801291: 
2025-07-17T11:07:26.0537422Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0537637Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0537845Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0538047Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0538545Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0538764Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0538975Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0538981Z 
2025-07-17T11:07:26.0538985Z 
2025-07-17T11:07:26.0539084Z SIGABRT(6), PID: 1800821, Thread 1801108: 
2025-07-17T11:07:26.0539553Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0539769Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0539985Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0540183Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0540686Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0540986Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0541200Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0541296Z SIGABRT(6), PID: 1800825, Thread 1801294: 
2025-07-17T11:07:26.0541775Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0542055Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0542267Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0548446Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0549182Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0549432Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0549663Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0549770Z 
2025-07-17T11:07:26.0549774Z 
2025-07-17T11:07:26.0549885Z SIGABRT(6), PID: 1800825, Thread 1801297: 
2025-07-17T11:07:26.0550403Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0550622Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0550836Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0551045Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0551564Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0551772Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0551990Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0551995Z 
2025-07-17T11:07:26.0552091Z SIGABRT(6), PID: 1800821, Thread 1801112: 
2025-07-17T11:07:26.0552578Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0552788Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0552999Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0553199Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0553702Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0553912Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0554128Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0554133Z 
2025-07-17T11:07:26.0554224Z SIGABRT(6), PID: 1800825, Thread 1801299: 
2025-07-17T11:07:26.0554704Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0554917Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0555131Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0555398Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0555897Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0556104Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0556371Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0556377Z 
2025-07-17T11:07:26.0556470Z SIGABRT(6), PID: 1800821, Thread 1801116: 
2025-07-17T11:07:26.0556944Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0557207Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0557426Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0557630Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0558124Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0558396Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0558606Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0558610Z 
2025-07-17T11:07:26.0558704Z SIGABRT(6), PID: 1800821, Thread 1801120: 
2025-07-17T11:07:26.0559175Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0559387Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0559593Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0559793Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0560289Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0560503Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0560709Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0560715Z 
2025-07-17T11:07:26.0560809Z SIGABRT(6), PID: 1800825, Thread 1801301: 
2025-07-17T11:07:26.0561276Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0561487Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0561691Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0561895Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0562387Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0562599Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0562805Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0562809Z 
2025-07-17T11:07:26.0562903Z SIGABRT(6), PID: 1800821, Thread 1801125: 
2025-07-17T11:07:26.0563370Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0563657Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0563857Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0564063Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0564617Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0564832Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0565039Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0565100Z 
2025-07-17T11:07:26.0565195Z SIGABRT(6), PID: 1800825, Thread 1801303: 
2025-07-17T11:07:26.0565659Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0565869Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0566143Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0566350Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0566841Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0567053Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0567261Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0567265Z 
2025-07-17T11:07:26.0567362Z SIGABRT(6), PID: 1800821, Thread 1801130: 
2025-07-17T11:07:26.0567833Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0568043Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0568253Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0568453Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0568952Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0569162Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0569377Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0569383Z 
2025-07-17T11:07:26.0569474Z SIGABRT(6), PID: 1800825, Thread 1801306: 
2025-07-17T11:07:26.0569949Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0570157Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0570366Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0570564Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0571059Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0571267Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0571551Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0571556Z 
2025-07-17T11:07:26.0571646Z SIGABRT(6), PID: 1800821, Thread 1801136: 
2025-07-17T11:07:26.0572115Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0572323Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0572588Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0572790Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0573371Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0573585Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0573800Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0573805Z 
2025-07-17T11:07:26.0573894Z SIGABRT(6), PID: 1800825, Thread 1801308: 
2025-07-17T11:07:26.0574441Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0574652Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0574860Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0575062Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0575577Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0575785Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0576079Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0576087Z 
2025-07-17T11:07:26.0576180Z SIGABRT(6), PID: 1800821, Thread 1801141: 
2025-07-17T11:07:26.0576653Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0576859Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0577068Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0577268Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0577765Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0577971Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0578182Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0578187Z 
2025-07-17T11:07:26.0578280Z SIGABRT(6), PID: 1800825, Thread 1801310: 
2025-07-17T11:07:26.0578751Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0578962Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0579167Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0579368Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0579948Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0580157Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0580369Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0580373Z 
2025-07-17T11:07:26.0580466Z SIGABRT(6), PID: 1800825, Thread 1801312: 
2025-07-17T11:07:26.0580999Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0581215Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0581510Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0581730Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0582255Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0582550Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0582763Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0582767Z 
2025-07-17T11:07:26.0582867Z SIGABRT(6), PID: 1800821, Thread 1801145: 
2025-07-17T11:07:26.0583352Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0583565Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0583771Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0583980Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0584499Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0584719Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0584929Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0584933Z 
2025-07-17T11:07:26.0585027Z SIGABRT(6), PID: 1800825, Thread 1801314: 
2025-07-17T11:07:26.0585505Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0585717Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0585923Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0586128Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0586629Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0586842Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0587052Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0587056Z 
2025-07-17T11:07:26.0587152Z SIGABRT(6), PID: 1800821, Thread 1801149: 
2025-07-17T11:07:26.0587623Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0587833Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0588114Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0588318Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0588816Z frame #4: <unknown function> + 0x3475ab (0x7fc2624375ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0589081Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0589297Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0589392Z SIGABRT(6), PID: 1800825, Thread 1801316: 
2025-07-17T11:07:26.0589911Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0590128Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0590331Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0590532Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0591084Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0591296Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0591509Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0591513Z 
2025-07-17T11:07:26.0591517Z 
2025-07-17T11:07:26.0591604Z SIGABRT(6), PID: 1800821, Thread 1801340: 
2025-07-17T11:07:26.0592076Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0592528Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fc173c9508a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0592739Z frame #2: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0592930Z frame #3: pthread_kill + 0x12c (0x7fc2655a19fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0593105Z frame #4: raise + 0x16 (0x7fc26554d476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0593265Z frame #5: abort + 0xd3 (0x7fc2655337f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0593510Z frame #6: <unknown function> + 0x972cc (0x7fc0e181e2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0593747Z frame #7: <unknown function> + 0x95f47 (0x7fc0e181cf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0593980Z frame #8: <unknown function> + 0x34b71 (0x7fc0e17bbb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0594197Z frame #9: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0594417Z frame #10: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0594423Z 
2025-07-17T11:07:26.0594516Z SIGABRT(6), PID: 1800825, Thread 1801317: 
2025-07-17T11:07:26.0595000Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0595206Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0595414Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0595614Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0596119Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0596403Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0596621Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0596626Z 
2025-07-17T11:07:26.0596715Z SIGABRT(6), PID: 1800825, Thread 1801318: 
2025-07-17T11:07:26.0597250Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0597456Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0597675Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0597928Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0598448Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0598667Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0598949Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0598954Z 
2025-07-17T11:07:26.0599044Z SIGABRT(6), PID: 1800821, Thread 1804268: 
2025-07-17T11:07:26.0599530Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0599743Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0599918Z frame #2: __poll + 0x4f (0x7fc265623bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0600334Z frame #3: <unknown function> + 0x489f (0x7fc26512f89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.0600568Z frame #4: <unknown function> + 0x1483bc (0x55c5354c53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0600815Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0601046Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0601286Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0601513Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0601760Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55c5354ada65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0601995Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0602243Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c5354abc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0602475Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0602709Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0602937Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0603170Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0603392Z frame #16: <unknown function> + 0x147734 (0x55c5354c4734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0603617Z frame #17: <unknown function> + 0x21ca82 (0x55c535599a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0603843Z frame #18: <unknown function> + 0x21c9e4 (0x55c5355999e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0604056Z frame #19: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0604348Z frame #20: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0604353Z 
2025-07-17T11:07:26.0604444Z SIGABRT(6), PID: 1800825, Thread 1801320: 
2025-07-17T11:07:26.0604930Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0605196Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0605414Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0605615Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0606179Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0606395Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0606608Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0606671Z 
2025-07-17T11:07:26.0606767Z SIGABRT(6), PID: 1800825, Thread 1801321: 
2025-07-17T11:07:26.0607239Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0607451Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0607654Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0607856Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0608352Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0608568Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0608777Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0608781Z 
2025-07-17T11:07:26.0608874Z SIGABRT(6), PID: 1800825, Thread 1801322: 
2025-07-17T11:07:26.0609342Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0609553Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0609754Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0609954Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0610447Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0610662Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0610875Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0610880Z 
2025-07-17T11:07:26.0610974Z SIGABRT(6), PID: 1800821, Thread 1804269: 
2025-07-17T11:07:26.0611438Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0611649Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0611853Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0612079Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fc26559f2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0612598Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fc19f9dd048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0613028Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fc19f9dfbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0613343Z frame #6: <unknown function> + 0xdc253 (0x7fc171f01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0613559Z frame #7: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0613768Z frame #8: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0613772Z 
2025-07-17T11:07:26.0613868Z SIGABRT(6), PID: 1800825, Thread 1801323: 
2025-07-17T11:07:26.0614393Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0614607Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0614813Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0615076Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0615574Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0615784Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0616045Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0616049Z 
2025-07-17T11:07:26.0616150Z SIGABRT(6), PID: 1800825, Thread 1801324: 
2025-07-17T11:07:26.0616623Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0616833Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0617037Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0617237Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0617736Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0617947Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0618161Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0618166Z 
2025-07-17T11:07:26.0618258Z SIGABRT(6), PID: 1800821, Thread 1804270: 
2025-07-17T11:07:26.0618725Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0618928Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0619135Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0619359Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fc26559f2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0619838Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fc19f9d7741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0620067Z frame #5: <unknown function> + 0xdc253 (0x7fc171f01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0620277Z frame #6: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0620569Z frame #7: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0620573Z 
2025-07-17T11:07:26.0620665Z SIGABRT(6), PID: 1800825, Thread 1801325: 
2025-07-17T11:07:26.0621132Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0621337Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0621609Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0621816Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0622378Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0622594Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0622804Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0622808Z 
2025-07-17T11:07:26.0622973Z SIGABRT(6), PID: 1800821, Thread 1804273: 
2025-07-17T11:07:26.0623443Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0623651Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0623818Z frame #2: ioctl + 0x3f (0x7fc26562594f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0624056Z frame #3: <unknown function> + 0x13eef0 (0x7fc0e18c5ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0624291Z frame #4: <unknown function> + 0x1378c4 (0x7fc0e18be8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0624523Z frame #5: <unknown function> + 0x95b2b (0x7fc0e181cb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0624749Z frame #6: <unknown function> + 0x34b71 (0x7fc0e17bbb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0624959Z frame #7: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0625166Z frame #8: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0625170Z 
2025-07-17T11:07:26.0625265Z SIGABRT(6), PID: 1800825, Thread 1801326: 
2025-07-17T11:07:26.0625737Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0625950Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0626154Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0626358Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0626853Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0627066Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0627275Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0627281Z 
2025-07-17T11:07:26.0627376Z SIGABRT(6), PID: 1800821, Thread 1805097: 
2025-07-17T11:07:26.0627839Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0628052Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0628220Z frame #2: __poll + 0x4f (0x7fc265623bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0628504Z frame #3: <unknown function> + 0xd4540 (0x7fc129a35540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0628709Z frame #4: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0628920Z frame #5: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0628924Z 
2025-07-17T11:07:26.0629016Z SIGABRT(6), PID: 1800825, Thread 1801327: 
2025-07-17T11:07:26.0629542Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0629754Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0630014Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0630220Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0630713Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0630926Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0631195Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0631199Z 
2025-07-17T11:07:26.0631295Z SIGABRT(6), PID: 1800821, Thread 1805104: 
2025-07-17T11:07:26.0631758Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0631966Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0632130Z frame #2: __poll + 0x4f (0x7fc265623bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0632352Z frame #3: <unknown function> + 0xd65a2 (0x7fc129a375a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0632556Z frame #4: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0632764Z frame #5: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0632770Z 
2025-07-17T11:07:26.0632858Z SIGABRT(6), PID: 1800825, Thread 1801328: 
2025-07-17T11:07:26.0633330Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0633533Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0633738Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0633938Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0634436Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0634643Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0634855Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0634859Z 
2025-07-17T11:07:26.0634948Z SIGABRT(6), PID: 1800821, Thread 1805109: 
2025-07-17T11:07:26.0635421Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0635626Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0635834Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0636031Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0636317Z frame #4: <unknown function> + 0xd1e7f (0x7fc129a32e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0636524Z frame #5: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0636736Z frame #6: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0636740Z 
2025-07-17T11:07:26.0636829Z SIGABRT(6), PID: 1800825, Thread 1801329: 
2025-07-17T11:07:26.0637357Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0637566Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0637842Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0638041Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0638541Z frame #4: <unknown function> + 0x3475ab (0x7faa0255f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0638748Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0639019Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0639023Z 
2025-07-17T11:07:26.0639113Z SIGABRT(6), PID: 1800825, Thread 1801348: 
2025-07-17T11:07:26.0639592Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0640045Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fa913dbe08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0640262Z frame #2: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0640459Z frame #3: pthread_kill + 0x12c (0x7faa056c99fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0640627Z frame #4: raise + 0x16 (0x7faa05675476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0640790Z frame #5: abort + 0xd3 (0x7faa0565b7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0641030Z frame #6: <unknown function> + 0x972cc (0x7fa8819472cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0641262Z frame #7: <unknown function> + 0x95f47 (0x7fa881945f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0641497Z frame #8: <unknown function> + 0x34b71 (0x7fa8818e4b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0641704Z frame #9: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0641926Z frame #10: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0641932Z 
2025-07-17T11:07:26.0642023Z SIGABRT(6), PID: 1800821, Thread 1805121: 
2025-07-17T11:07:26.0642505Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0642719Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0642924Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0643140Z frame #3: <unknown function> + 0x9ca53 (0x7fc2655a7a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0643394Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55c535491bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0643627Z frame #5: <unknown function> + 0x1d314f (0x55c53555014f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0643853Z frame #6: <unknown function> + 0x1d2f6b (0x55c53554ff6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0644166Z frame #7: <unknown function> + 0x6df36 (0x55c5353eaf36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0644414Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0644643Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0644889Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0645174Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0645423Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0645647Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0645954Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0646181Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0646418Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0646636Z frame #17: <unknown function> + 0x147734 (0x55c5354c4734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0646921Z frame #18: <unknown function> + 0x21ca82 (0x55c535599a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0647143Z frame #19: <unknown function> + 0x21c9e4 (0x55c5355999e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0647363Z frame #20: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0647575Z frame #21: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0647585Z 
2025-07-17T11:07:26.0647677Z SIGABRT(6), PID: 1800825, Thread 1804066: 
2025-07-17T11:07:26.0648153Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0648365Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0648536Z frame #2: __poll + 0x4f (0x7faa0574bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0648944Z frame #3: <unknown function> + 0x489f (0x7faa0525789f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.0649176Z frame #4: <unknown function> + 0x1483bc (0x561c0fb363bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0649418Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0649654Z frame #6: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0649892Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0650129Z frame #8: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0650377Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x561c0fb1ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0650615Z frame #10: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0650864Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x561c0fb1cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0651108Z frame #12: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0651346Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0651575Z frame #14: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0651811Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0652112Z frame #16: <unknown function> + 0x147734 (0x561c0fb35734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0652332Z frame #17: <unknown function> + 0x21ca82 (0x561c0fc0aa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0652559Z frame #18: <unknown function> + 0x21c9e4 (0x561c0fc0a9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0652776Z frame #19: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0653067Z frame #20: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0653072Z 
2025-07-17T11:07:26.0653167Z SIGABRT(6), PID: 1800821, Thread 1805139: 
2025-07-17T11:07:26.0653646Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0653918Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0654098Z frame #2: read + 0x4c (0x7fc26561f81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0654281Z frame #3: _Py_read + 0x4c (0x55c535557a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0654509Z frame #4: <unknown function> + 0x208c3b (0x55c535585c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0654790Z frame #5: <unknown function> + 0x14648d (0x55c5354c348d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0655042Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55c5354d2215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0655259Z frame #7: <unknown function> + 0x208ae7 (0x55c535585ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0655477Z frame #8: <unknown function> + 0x208472 (0x55c535585472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0655699Z frame #9: <unknown function> + 0x1da6ee (0x55c5355576ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0655997Z frame #10: <unknown function> + 0x1483bc (0x55c5354c53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0656244Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0656475Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0656711Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55c5354a93bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0656943Z frame #14: <unknown function> + 0x147734 (0x55c5354c4734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0657193Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55c5354abc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0657425Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0657660Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0657892Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55c5354b90ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0658128Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55c5354a97b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0658354Z frame #20: <unknown function> + 0x147734 (0x55c5354c4734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0658577Z frame #21: <unknown function> + 0x21ca82 (0x55c535599a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0658802Z frame #22: <unknown function> + 0x21c9e4 (0x55c5355999e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0659019Z frame #23: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0659240Z frame #24: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0659244Z 
2025-07-17T11:07:26.0659340Z SIGABRT(6), PID: 1800825, Thread 1804067: 
2025-07-17T11:07:26.0659827Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0660240Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0660460Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0660689Z frame #3: pthread_cond_clockwait + 0x1fd (0x7faa056c72dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0661216Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fa93fb06048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0661645Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fa93fb08bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0661884Z frame #6: <unknown function> + 0xdc253 (0x7fa91202a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0662164Z frame #7: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0662394Z frame #8: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0662398Z 
2025-07-17T11:07:26.0662495Z SIGABRT(6), PID: 1800825, Thread 1804068: 
2025-07-17T11:07:26.0663047Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0663256Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0663469Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0663694Z frame #3: pthread_cond_clockwait + 0x1fd (0x7faa056c72dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0664179Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fa93fb00741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0664411Z frame #5: <unknown function> + 0xdc253 (0x7fa91202a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0664629Z frame #6: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0664846Z frame #7: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0664851Z 
2025-07-17T11:07:26.0664953Z SIGABRT(6), PID: 1800825, Thread 1804072: 
2025-07-17T11:07:26.0665421Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0665637Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0665805Z frame #2: ioctl + 0x3f (0x7faa0574d94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0666045Z frame #3: <unknown function> + 0x13eef0 (0x7fa8819eeef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0666275Z frame #4: <unknown function> + 0x1378c4 (0x7fa8819e78c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0666515Z frame #5: <unknown function> + 0x95b2b (0x7fa881945b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0666743Z frame #6: <unknown function> + 0x34b71 (0x7fa8818e4b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0666960Z frame #7: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0667171Z frame #8: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0667175Z 
2025-07-17T11:07:26.0667276Z SIGABRT(6), PID: 1800821, Thread 1805195: 
2025-07-17T11:07:26.0667743Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0667960Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0668171Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0668450Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0668833Z frame #4: <unknown function> + 0x5a02023 (0x7fc1a99cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0669471Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc1a99cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0670070Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc1a99c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0670522Z frame #7: <unknown function> + 0x79d655 (0x7fc1baeb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0670778Z frame #8: <unknown function> + 0xdc253 (0x7fc171f01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0670996Z frame #9: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0671299Z frame #10: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0671304Z 
2025-07-17T11:07:26.0671404Z SIGABRT(6), PID: 1800825, Thread 1805096: 
2025-07-17T11:07:26.0671895Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0672109Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0672285Z frame #2: __poll + 0x4f (0x7faa0574bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0672510Z frame #3: <unknown function> + 0xd4540 (0x7fa8c9b5e540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0672728Z frame #4: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0672937Z frame #5: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0672941Z 
2025-07-17T11:07:26.0673046Z SIGABRT(6), PID: 1800825, Thread 1805103: 
2025-07-17T11:07:26.0673522Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0673735Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0673900Z frame #2: __poll + 0x4f (0x7faa0574bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0674125Z frame #3: <unknown function> + 0xd65a2 (0x7fa8c9b605a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0674337Z frame #4: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0674551Z frame #5: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0674557Z 
2025-07-17T11:07:26.0674649Z SIGABRT(6), PID: 1800825, Thread 1805111: 
2025-07-17T11:07:26.0675123Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0675331Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0675543Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0675749Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0675975Z frame #4: <unknown function> + 0xd1e7f (0x7fa8c9b5be7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.0676186Z frame #5: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0676402Z frame #6: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0676472Z 
2025-07-17T11:07:26.0676567Z SIGABRT(6), PID: 1800825, Thread 1805122: 
2025-07-17T11:07:26.0677042Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0677251Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0677523Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0677741Z frame #3: <unknown function> + 0x9ca53 (0x7faa056cfa53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0678013Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x561c0fb02bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0678300Z frame #5: <unknown function> + 0x1d314f (0x561c0fbc114f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0678537Z frame #6: <unknown function> + 0x1d2f6b (0x561c0fbc0f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0678763Z frame #7: <unknown function> + 0x6df36 (0x561c0fa5bf36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0679032Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0679336Z frame #9: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0679606Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0679838Z frame #11: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0680084Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0680316Z frame #13: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0680560Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0680789Z frame #15: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0681030Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0681254Z frame #17: <unknown function> + 0x147734 (0x561c0fb35734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0681484Z frame #18: <unknown function> + 0x21ca82 (0x561c0fc0aa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0681704Z frame #19: <unknown function> + 0x21c9e4 (0x561c0fc0a9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0681924Z frame #20: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0682141Z frame #21: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0682145Z 
2025-07-17T11:07:26.0682247Z SIGABRT(6), PID: 1800825, Thread 1805138: 
2025-07-17T11:07:26.0682728Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0682950Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0683119Z frame #2: read + 0x4c (0x7faa0574781c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0683312Z frame #3: _Py_read + 0x4c (0x561c0fbc8a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0683542Z frame #4: <unknown function> + 0x208c3b (0x561c0fbf6c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0683777Z frame #5: <unknown function> + 0x14648d (0x561c0fb3448d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0684025Z frame #6: PyObject_VectorcallMethod + 0x85 (0x561c0fb43215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0684259Z frame #7: <unknown function> + 0x208ae7 (0x561c0fbf6ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0684546Z frame #8: <unknown function> + 0x208472 (0x561c0fbf6472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0684775Z frame #9: <unknown function> + 0x1da6ee (0x561c0fbc86ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0685004Z frame #10: <unknown function> + 0x1483bc (0x561c0fb363bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0685258Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0685542Z frame #12: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0685794Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x561c0fb1a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0686012Z frame #14: <unknown function> + 0x147734 (0x561c0fb35734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0686330Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x561c0fb1cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0686561Z frame #16: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0686804Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0687089Z frame #18: _PyFunction_Vectorcall + 0x6c (0x561c0fb2a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0687330Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x561c0fb1a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0687549Z frame #20: <unknown function> + 0x147734 (0x561c0fb35734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0687774Z frame #21: <unknown function> + 0x21ca82 (0x561c0fc0aa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0687996Z frame #22: <unknown function> + 0x21c9e4 (0x561c0fc0a9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0688226Z frame #23: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0688448Z frame #24: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0688452Z 
2025-07-17T11:07:26.0688556Z SIGABRT(6), PID: 1800821, Thread 1805196: 
2025-07-17T11:07:26.0689032Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0689256Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0689468Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0689678Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0690059Z frame #4: <unknown function> + 0x5a02023 (0x7fc1a99cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0690634Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc1a99cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0691236Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc1a99c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0691630Z frame #7: <unknown function> + 0x79d655 (0x7fc1baeb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0691866Z frame #8: <unknown function> + 0xdc253 (0x7fc171f01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0692083Z frame #9: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0692299Z frame #10: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0692312Z 
2025-07-17T11:07:26.0692407Z SIGABRT(6), PID: 1800825, Thread 1805172: 
2025-07-17T11:07:26.0692954Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0693163Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0693376Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0693582Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0694028Z frame #4: <unknown function> + 0x5a02023 (0x7fa949af4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0694652Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa949af81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0695259Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa949aef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0695647Z frame #7: <unknown function> + 0x79d655 (0x7fa95afdf655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0696004Z frame #8: <unknown function> + 0xdc253 (0x7fa91202a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0696223Z frame #9: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0696448Z frame #10: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0696452Z 
2025-07-17T11:07:26.0696544Z SIGABRT(6), PID: 1800821, Thread 1805198: 
2025-07-17T11:07:26.0697029Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0697238Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0697447Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0697647Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0698030Z frame #4: <unknown function> + 0x5a02023 (0x7fc1a99cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0698597Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc1a99cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0699195Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc1a99c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0699577Z frame #7: <unknown function> + 0x79d655 (0x7fc1baeb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0699816Z frame #8: <unknown function> + 0xdc253 (0x7fc171f01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0700029Z frame #9: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0700256Z frame #10: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0700260Z 
2025-07-17T11:07:26.0700357Z SIGABRT(6), PID: 1800825, Thread 1805173: 
2025-07-17T11:07:26.0700840Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0701051Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0701267Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0701552Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0701930Z frame #4: <unknown function> + 0x5a02023 (0x7fa949af4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0702503Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa949af81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0703169Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa949aef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0703650Z frame #7: <unknown function> + 0x79d655 (0x7fa95afdf655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0703884Z frame #8: <unknown function> + 0xdc253 (0x7fa91202a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0704104Z frame #9: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0704319Z frame #10: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0704392Z 
2025-07-17T11:07:26.0704493Z SIGABRT(6), PID: 1800821, Thread 1805199: 
2025-07-17T11:07:26.0704964Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0705178Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0705386Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0705597Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0705971Z frame #4: <unknown function> + 0x5a02023 (0x7fc1a99cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0706536Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc1a99cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0707128Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc1a99c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0707512Z frame #7: <unknown function> + 0x79d655 (0x7fc1baeb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0707741Z frame #8: <unknown function> + 0xdc253 (0x7fc171f01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0707958Z frame #9: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0708182Z frame #10: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0708186Z 
2025-07-17T11:07:26.0708286Z SIGABRT(6), PID: 1800825, Thread 1805175: 
2025-07-17T11:07:26.0708757Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0708973Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0709179Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0709386Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0709756Z frame #4: <unknown function> + 0x5a02023 (0x7fa949af4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0710322Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa949af81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0710992Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa949aef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0711375Z frame #7: <unknown function> + 0x79d655 (0x7fa95afdf655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0711664Z frame #8: <unknown function> + 0xdc253 (0x7fa91202a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0711878Z frame #9: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0712100Z frame #10: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0712161Z 
2025-07-17T11:07:26.0712257Z SIGABRT(6), PID: 1800821, Thread 1805200: 
2025-07-17T11:07:26.0712742Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0712952Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0713229Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0713431Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0713814Z frame #4: <unknown function> + 0x5a02023 (0x7fc1a99cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0714373Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc1a99cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0714964Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc1a99c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0715346Z frame #7: <unknown function> + 0x79d655 (0x7fc1baeb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0715578Z frame #8: <unknown function> + 0xdc253 (0x7fc171f01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0715789Z frame #9: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0716011Z frame #10: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0716015Z 
2025-07-17T11:07:26.0716106Z SIGABRT(6), PID: 1800825, Thread 1805177: 
2025-07-17T11:07:26.0716584Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0716794Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0717007Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0717210Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0717586Z frame #4: <unknown function> + 0x5a02023 (0x7fa949af4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0718155Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa949af81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0718751Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa949aef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0719200Z frame #7: <unknown function> + 0x79d655 (0x7fa95afdf655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0719437Z frame #8: <unknown function> + 0xdc253 (0x7fa91202a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0719649Z frame #9: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0719869Z frame #10: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0719873Z 
2025-07-17T11:07:26.0720022Z SIGABRT(6), PID: 1800821, Thread 1805201: 
2025-07-17T11:07:26.0720505Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0720721Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0720986Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0721194Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0721563Z frame #4: <unknown function> + 0x5a02023 (0x7fc1a99cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0722189Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc1a99cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0722781Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc1a99c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0723173Z frame #7: <unknown function> + 0x79d655 (0x7fc1baeb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0723403Z frame #8: <unknown function> + 0xdc253 (0x7fc171f01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0723617Z frame #9: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0723829Z frame #10: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0723835Z 
2025-07-17T11:07:26.0723932Z SIGABRT(6), PID: 1800825, Thread 1805180: 
2025-07-17T11:07:26.0724405Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0724622Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0724828Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0725035Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0725405Z frame #4: <unknown function> + 0x5a02023 (0x7fa949af4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0725968Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa949af81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0726562Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa949aef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0726957Z frame #7: <unknown function> + 0x79d655 (0x7fa95afdf655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0727184Z frame #8: <unknown function> + 0xdc253 (0x7fa91202a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0727400Z frame #9: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0727687Z frame #10: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0727691Z 
2025-07-17T11:07:26.0727787Z SIGABRT(6), PID: 1800821, Thread 1805202: 
2025-07-17T11:07:26.0728261Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0728484Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0728752Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0728962Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0729336Z frame #4: <unknown function> + 0x5a02023 (0x7fc1a99cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0729959Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc1a99cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0730561Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc1a99c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0731000Z frame #7: <unknown function> + 0x79d655 (0x7fc1baeb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0731237Z frame #8: <unknown function> + 0xdc253 (0x7fc171f01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0731446Z frame #9: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0731665Z frame #10: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0731670Z 
2025-07-17T11:07:26.0731762Z SIGABRT(6), PID: 1800825, Thread 1805183: 
2025-07-17T11:07:26.0732246Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0732454Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0732668Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0732870Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0733254Z frame #4: <unknown function> + 0x5a02023 (0x7fa949af4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0733817Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa949af81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0734414Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa949aef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0734798Z frame #7: <unknown function> + 0x79d655 (0x7fa95afdf655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0735034Z frame #8: <unknown function> + 0xdc253 (0x7fa91202a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0735248Z frame #9: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0735473Z frame #10: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0735477Z 
2025-07-17T11:07:26.0735568Z SIGABRT(6), PID: 1800821, Thread 1805203: 
2025-07-17T11:07:26.0736139Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fc173c94b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0736462Z frame #1: <unknown function> + 0x42520 (0x7fc26554d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0736682Z frame #2: <unknown function> + 0x91117 (0x7fc26559c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0736889Z frame #3: pthread_cond_wait + 0x211 (0x7fc26559ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0737269Z frame #4: <unknown function> + 0x5a02023 (0x7fc1a99cb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0737921Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fc1a99cf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0738594Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fc1a99c61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0738987Z frame #7: <unknown function> + 0x79d655 (0x7fc1baeb6655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0739221Z frame #8: <unknown function> + 0xdc253 (0x7fc171f01253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0739512Z frame #9: <unknown function> + 0x94ac3 (0x7fc26559fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0739730Z frame #10: <unknown function> + 0x126850 (0x7fc265631850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0739735Z 
2025-07-17T11:07:26.0739836Z SIGABRT(6), PID: 1800825, Thread 1805185: 
2025-07-17T11:07:26.0740320Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0740541Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0740750Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0740961Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0741330Z frame #4: <unknown function> + 0x5a02023 (0x7fa949af4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0741892Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa949af81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0742487Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa949aef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0742878Z frame #7: <unknown function> + 0x79d655 (0x7fa95afdf655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0743109Z frame #8: <unknown function> + 0xdc253 (0x7fa91202a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0743330Z frame #9: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0743548Z frame #10: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0743554Z 
2025-07-17T11:07:26.0743652Z SIGABRT(6), PID: 1800819, Thread 1800819: 
2025-07-17T11:07:26.0744126Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0744342Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0744577Z frame #2: <unknown function> + 0x7743a (0x7fa33844f43a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0744815Z frame #3: <unknown function> + 0x76c8e (0x7fa33844ec8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0745042Z frame #4: <unknown function> + 0x6ad69 (0x7fa338442d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0745343Z frame #5: <unknown function> + 0x3f54db (0x7fa3c90a74db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0745558Z frame #6: <unknown function> + 0x3e11d8 (0x7fa3c90931d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0745782Z frame #7: <unknown function> + 0xb2f86 (0x7fa3c8d64f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0745992Z frame #8: <unknown function> + 0x8e501 (0x7fa3c8d40501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0746444Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fa3f3b477ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.0746837Z frame #10: <unknown function> + 0xd307d9 (0x7fa41209a7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0747132Z frame #11: <unknown function> + 0x139902 (0x555ec3514902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0747391Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x555ec350ba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0747628Z frame #13: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0747940Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x555ec350ba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0748172Z frame #15: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0748421Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0748655Z frame #17: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0748894Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0749126Z frame #19: <unknown function> + 0x1474b2 (0x555ec35224b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0749334Z frame #20: PyObject_Call + 0xbc (0x555ec3522e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0749570Z frame #21: <unknown function> + 0x1ff69d (0x555ec35da69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0749794Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x555ec3510023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0750046Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x555ec350bcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0750272Z frame #24: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0750516Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x555ec35073bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0750736Z frame #26: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0750979Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x555ec35073bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0751204Z frame #28: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0751453Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x555ec350ba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0751711Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x555ec350f4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0751937Z frame #31: _PyObject_Call_Prepend + 0x69 (0x555ec3520ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0752161Z frame #32: <unknown function> + 0x205949 (0x555ec35e0949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0752391Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x555ec3510023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0752635Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x555ec350bcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0752870Z frame #35: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0753103Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x555ec35073bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0753404Z frame #37: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0753638Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x555ec35073bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0753870Z frame #39: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0754167Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x555ec35073bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0754402Z frame #41: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0754645Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x555ec35083f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0754931Z frame #43: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0755326Z frame #44: <unknown function> + 0x7ab3f7 (0x7fa411b153f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0755559Z frame #45: <unknown function> + 0x13bc88 (0x555ec3516c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0755766Z frame #46: PyObject_Call + 0x20f (0x555ec3522f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0756079Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x555ec350c98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0756302Z frame #48: <unknown function> + 0x147608 (0x555ec3522608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0756550Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x555ec3509c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0756777Z frame #50: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0757020Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x555ec35073bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0757244Z frame #52: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0757490Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x555ec35083f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0757717Z frame #54: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0757961Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x555ec35073bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0758192Z frame #56: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0758433Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x555ec3509c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0758657Z frame #58: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0758903Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x555ec3509c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0759123Z frame #60: <unknown function> + 0x147608 (0x555ec3522608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0759373Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x555ec3509c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0759591Z frame #62: <unknown function> + 0x147608 (0x555ec3522608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0759597Z 
2025-07-17T11:07:26.0759697Z SIGABRT(6), PID: 1800819, Thread 1800859: 
2025-07-17T11:07:26.0760176Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0760398Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0760610Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0760820Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0761333Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0761623Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0761842Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0761846Z 
2025-07-17T11:07:26.0761946Z SIGABRT(6), PID: 1800819, Thread 1800861: 
2025-07-17T11:07:26.0762477Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0762702Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0762966Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0763178Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0763683Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0763958Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0764173Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0764184Z 
2025-07-17T11:07:26.0764277Z SIGABRT(6), PID: 1800819, Thread 1800862: 
2025-07-17T11:07:26.0764757Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0764966Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0765178Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0765380Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0765884Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0766094Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0766313Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0766319Z 
2025-07-17T11:07:26.0766409Z SIGABRT(6), PID: 1800819, Thread 1800864: 
2025-07-17T11:07:26.0766887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0767097Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0767306Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0767507Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0768011Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0768236Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0768591Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0768595Z 
2025-07-17T11:07:26.0768737Z SIGABRT(6), PID: 1800825, Thread 1805188: 
2025-07-17T11:07:26.0769213Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa913dbdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0769496Z frame #1: <unknown function> + 0x42520 (0x7faa05675520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0769831Z frame #2: <unknown function> + 0x91117 (0x7faa056c4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0770105Z frame #3: pthread_cond_wait + 0x211 (0x7faa056c6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0770533Z frame #4: <unknown function> + 0x5a02023 (0x7fa949af4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0771202Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa949af81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0771833Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa949aef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.0772288Z frame #7: <unknown function> + 0x79d655 (0x7fa95afdf655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0772528Z frame #8: <unknown function> + 0xdc253 (0x7fa91202a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0772770Z frame #9: <unknown function> + 0x94ac3 (0x7faa056c7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0773076Z frame #10: <unknown function> + 0x126850 (0x7faa05759850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0773195Z SIGABRT(6), PID: 1800819, Thread 1800866: 
2025-07-17T11:07:26.0773733Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0773972Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0774184Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0774388Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0774903Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0775143Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0775360Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0775364Z 
2025-07-17T11:07:26.0775369Z 
2025-07-17T11:07:26.0775465Z SIGABRT(6), PID: 1800819, Thread 1800868: 
2025-07-17T11:07:26.0776017Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0776227Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0776442Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0776643Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0777148Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0777357Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0777573Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0777577Z 
2025-07-17T11:07:26.0777670Z SIGABRT(6), PID: 1800819, Thread 1800871: 
2025-07-17T11:07:26.0778141Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0778364Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0778661Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0778869Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0779363Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0779580Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0779861Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0779866Z 
2025-07-17T11:07:26.0779965Z SIGABRT(6), PID: 1800819, Thread 1800873: 
2025-07-17T11:07:26.0780498Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0780717Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0780927Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0781128Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0781694Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0781916Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0782125Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0782130Z 
2025-07-17T11:07:26.0782226Z SIGABRT(6), PID: 1800819, Thread 1800875: 
2025-07-17T11:07:26.0782693Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0782914Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0783120Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0783324Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0783821Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0784037Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0784245Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0784250Z 
2025-07-17T11:07:26.0784349Z SIGABRT(6), PID: 1800819, Thread 1800877: 
2025-07-17T11:07:26.0784813Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0785030Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0785239Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0785446Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0785944Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0786158Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0786365Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0786369Z 
2025-07-17T11:07:26.0786468Z SIGABRT(6), PID: 1800819, Thread 1800879: 
2025-07-17T11:07:26.0787001Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0787217Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0787425Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0787628Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0788181Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0788412Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0788680Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0788689Z 
2025-07-17T11:07:26.0788784Z SIGABRT(6), PID: 1800819, Thread 1800881: 
2025-07-17T11:07:26.0789261Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0789630Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0789843Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0790047Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0790552Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0790762Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0790978Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0790984Z 
2025-07-17T11:07:26.0791076Z SIGABRT(6), PID: 1800819, Thread 1800883: 
2025-07-17T11:07:26.0791549Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0791761Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0791980Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0792182Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0792675Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0792887Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0793101Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0793106Z 
2025-07-17T11:07:26.0793196Z SIGABRT(6), PID: 1800819, Thread 1800885: 
2025-07-17T11:07:26.0793664Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0793872Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0794086Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0794290Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0794790Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0795000Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0795284Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0795288Z 
2025-07-17T11:07:26.0795380Z SIGABRT(6), PID: 1800819, Thread 1800887: 
2025-07-17T11:07:26.0795854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0796116Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0796337Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0796538Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0797092Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0797310Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0797523Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0797603Z 
2025-07-17T11:07:26.0797698Z SIGABRT(6), PID: 1800819, Thread 1800888: 
2025-07-17T11:07:26.0798183Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0798391Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0798602Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0798801Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0799302Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0799515Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0799728Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0799734Z 
2025-07-17T11:07:26.0799824Z SIGABRT(6), PID: 1800819, Thread 1800890: 
2025-07-17T11:07:26.0800300Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0800513Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0800721Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0800932Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0801431Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0801653Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0801864Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0801868Z 
2025-07-17T11:07:26.0801967Z SIGABRT(6), PID: 1800819, Thread 1800892: 
2025-07-17T11:07:26.0802437Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0802653Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0802860Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0803067Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0803633Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0803854Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0804065Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0804069Z 
2025-07-17T11:07:26.0804227Z SIGABRT(6), PID: 1800819, Thread 1800894: 
2025-07-17T11:07:26.0804697Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0804915Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0805180Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0805395Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0805893Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0806175Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0806385Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0806390Z 
2025-07-17T11:07:26.0806491Z SIGABRT(6), PID: 1800819, Thread 1800896: 
2025-07-17T11:07:26.0806959Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0807176Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0807383Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0807590Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0808085Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0808300Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0808511Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0808514Z 
2025-07-17T11:07:26.0808614Z SIGABRT(6), PID: 1800819, Thread 1800898: 
2025-07-17T11:07:26.0809082Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0809293Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0809501Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0809706Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0810202Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0810418Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0810630Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0810640Z 
2025-07-17T11:07:26.0810732Z SIGABRT(6), PID: 1800819, Thread 1800901: 
2025-07-17T11:07:26.0811215Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0811496Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0811715Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0811915Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0812416Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0812682Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0812905Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0812909Z 
2025-07-17T11:07:26.0813001Z SIGABRT(6), PID: 1800819, Thread 1800904: 
2025-07-17T11:07:26.0813542Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0813755Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0813966Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0814246Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0814750Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0814958Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0815176Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0815180Z 
2025-07-17T11:07:26.0815275Z SIGABRT(6), PID: 1800819, Thread 1800906: 
2025-07-17T11:07:26.0815758Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0816049Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0816265Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0816485Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0816986Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0817195Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0817410Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0817414Z 
2025-07-17T11:07:26.0817508Z SIGABRT(6), PID: 1800819, Thread 1800908: 
2025-07-17T11:07:26.0817981Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0818188Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0818405Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0818607Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0819104Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0819311Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0819531Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0819619Z 
2025-07-17T11:07:26.0819717Z SIGABRT(6), PID: 1800819, Thread 1800910: 
2025-07-17T11:07:26.0820194Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0820407Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0820620Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0820893Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0821397Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0821676Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0821896Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0821902Z 
2025-07-17T11:07:26.0821996Z SIGABRT(6), PID: 1800819, Thread 1800914: 
2025-07-17T11:07:26.0822462Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0822755Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0822967Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0823174Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0823672Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0823888Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0824103Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0824107Z 
2025-07-17T11:07:26.0824202Z SIGABRT(6), PID: 1800819, Thread 1800916: 
2025-07-17T11:07:26.0824673Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0824888Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0825095Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0825307Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0825803Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0826027Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0826238Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0826242Z 
2025-07-17T11:07:26.0826343Z SIGABRT(6), PID: 1800819, Thread 1800920: 
2025-07-17T11:07:26.0826813Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0827030Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0827234Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0827439Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0827937Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0828229Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0828443Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0828450Z 
2025-07-17T11:07:26.0828547Z SIGABRT(6), PID: 1800819, Thread 1800923: 
2025-07-17T11:07:26.0829073Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0829288Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0829493Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0829780Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0830278Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0830499Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0830766Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0830770Z 
2025-07-17T11:07:26.0830864Z SIGABRT(6), PID: 1800819, Thread 1800927: 
2025-07-17T11:07:26.0831337Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0831551Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0831760Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0831966Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0832461Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0832677Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0832885Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0832893Z 
2025-07-17T11:07:26.0832983Z SIGABRT(6), PID: 1800819, Thread 1800929: 
2025-07-17T11:07:26.0833454Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0833663Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0833876Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0834074Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0834571Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0834781Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0834994Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0835000Z 
2025-07-17T11:07:26.0835089Z SIGABRT(6), PID: 1800819, Thread 1800932: 
2025-07-17T11:07:26.0835562Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0835770Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0835981Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0836250Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0836746Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0836956Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0837230Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0837235Z 
2025-07-17T11:07:26.0837329Z SIGABRT(6), PID: 1800819, Thread 1800935: 
2025-07-17T11:07:26.0837807Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0838069Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0838288Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0838486Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0839044Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0839257Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0839469Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0839473Z 
2025-07-17T11:07:26.0839562Z SIGABRT(6), PID: 1800819, Thread 1800938: 
2025-07-17T11:07:26.0840041Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0840250Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0840464Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0840661Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0841163Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0841372Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0841585Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0841589Z 
2025-07-17T11:07:26.0841680Z SIGABRT(6), PID: 1800819, Thread 1800941: 
2025-07-17T11:07:26.0842155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0842363Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0842572Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0842772Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0843275Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0843491Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0843700Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0843704Z 
2025-07-17T11:07:26.0843802Z SIGABRT(6), PID: 1800819, Thread 1800944: 
2025-07-17T11:07:26.0844268Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0844550Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0844760Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0844965Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0845515Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0845731Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0845938Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0846000Z 
2025-07-17T11:07:26.0846098Z SIGABRT(6), PID: 1800819, Thread 1800947: 
2025-07-17T11:07:26.0846570Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0846783Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0847049Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0847253Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0847746Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0847958Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0848177Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0848182Z 
2025-07-17T11:07:26.0848278Z SIGABRT(6), PID: 1800824, Thread 1800824: 
2025-07-17T11:07:26.0848745Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0848957Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0849192Z frame #2: <unknown function> + 0x7743a (0x7f3b903e043a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0849425Z frame #3: <unknown function> + 0x76c8e (0x7f3b903dfc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0849651Z frame #4: <unknown function> + 0x6ad69 (0x7f3b903d3d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0849873Z frame #5: <unknown function> + 0x3f54db (0x7f3c210384db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0850084Z frame #6: <unknown function> + 0x3e11d8 (0x7f3c210241d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0850300Z frame #7: <unknown function> + 0xb2f86 (0x7f3c20cf5f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0850508Z frame #8: <unknown function> + 0x8e501 (0x7f3c20cd1501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.0850897Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f3c4bad87ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.0851291Z frame #10: <unknown function> + 0xd307d9 (0x7f3c6a02b7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0851522Z frame #11: <unknown function> + 0x139902 (0x559034963902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0851772Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55903495aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0852007Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0852250Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55903495aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0852552Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0852790Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0853023Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0853257Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0853542Z frame #19: <unknown function> + 0x1474b2 (0x5590349714b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0853751Z frame #20: PyObject_Call + 0xbc (0x559034971e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0853979Z frame #21: <unknown function> + 0x1ff69d (0x559034a2969d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0854260Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55903495f023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0854513Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55903495acc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0854737Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0855035Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5590349563bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0855257Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0855495Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5590349563bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0855716Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0856039Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55903495aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0856299Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55903495e4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0856522Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55903496fab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0856740Z frame #32: <unknown function> + 0x205949 (0x559034a2f949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0856969Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55903495f023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0857212Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55903495acc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0857443Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0857679Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5590349563bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0857905Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0858139Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5590349563bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0858369Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0858600Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5590349563bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0858829Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0859067Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5590349573f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0859292Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0859681Z frame #44: <unknown function> + 0x7ab3f7 (0x7f3c69aa63f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.0859911Z frame #45: <unknown function> + 0x13bc88 (0x559034965c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0860115Z frame #46: PyObject_Call + 0x20f (0x559034971f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0860446Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55903495b98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0860663Z frame #48: <unknown function> + 0x147608 (0x559034971608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0860909Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x559034958c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0861136Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0861461Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5590349563bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0861697Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0861939Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5590349573f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0862231Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0862476Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5590349563bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0862698Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0863011Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x559034958c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0863232Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0863473Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x559034958c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0863697Z frame #60: <unknown function> + 0x147608 (0x559034971608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0863939Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x559034958c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0864158Z frame #62: <unknown function> + 0x147608 (0x559034971608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0864164Z 
2025-07-17T11:07:26.0864262Z SIGABRT(6), PID: 1800819, Thread 1800950: 
2025-07-17T11:07:26.0864744Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0864967Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0865177Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0865386Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0865887Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0866104Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0866315Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0866321Z 
2025-07-17T11:07:26.0866419Z SIGABRT(6), PID: 1800824, Thread 1801128: 
2025-07-17T11:07:26.0866887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0867101Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0867304Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0867512Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0868013Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0868223Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0868503Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0868512Z 
2025-07-17T11:07:26.0868607Z SIGABRT(6), PID: 1800819, Thread 1800953: 
2025-07-17T11:07:26.0869091Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0869359Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0869576Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0869775Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0870338Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0870555Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0870772Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0870777Z 
2025-07-17T11:07:26.0870927Z SIGABRT(6), PID: 1800824, Thread 1801133: 
2025-07-17T11:07:26.0871399Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0871611Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0871819Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0872017Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0872517Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0872726Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0872939Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0872944Z 
2025-07-17T11:07:26.0873035Z SIGABRT(6), PID: 1800819, Thread 1800956: 
2025-07-17T11:07:26.0873510Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0873717Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0873930Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0874132Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0874636Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0874848Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0875063Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0875067Z 
2025-07-17T11:07:26.0875158Z SIGABRT(6), PID: 1800824, Thread 1801138: 
2025-07-17T11:07:26.0875636Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0875841Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0876049Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0876248Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0876815Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0877031Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0877243Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0877251Z 
2025-07-17T11:07:26.0877400Z SIGABRT(6), PID: 1800819, Thread 1800959: 
2025-07-17T11:07:26.0877880Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0878089Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0878365Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0878570Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0879071Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0879343Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0879560Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0879566Z 
2025-07-17T11:07:26.0879658Z SIGABRT(6), PID: 1800824, Thread 1801143: 
2025-07-17T11:07:26.0880123Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0880333Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0880537Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0880740Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0881232Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0881447Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0881655Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0881660Z 
2025-07-17T11:07:26.0881753Z SIGABRT(6), PID: 1800819, Thread 1800962: 
2025-07-17T11:07:26.0882218Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0882432Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0882639Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0882845Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0883338Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0883553Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0883760Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0883764Z 
2025-07-17T11:07:26.0883858Z SIGABRT(6), PID: 1800819, Thread 1800965: 
2025-07-17T11:07:26.0884325Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0884604Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0884813Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0885018Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0885510Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0885781Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0885992Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0885996Z 
2025-07-17T11:07:26.0886091Z SIGABRT(6), PID: 1800824, Thread 1801148: 
2025-07-17T11:07:26.0886616Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0886832Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0887032Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0887295Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0887792Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0888006Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0888214Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0888217Z 
2025-07-17T11:07:26.0888315Z SIGABRT(6), PID: 1800819, Thread 1800968: 
2025-07-17T11:07:26.0888785Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0889004Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0889209Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0889410Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0889905Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0890117Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0890325Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0890335Z 
2025-07-17T11:07:26.0890426Z SIGABRT(6), PID: 1800824, Thread 1801152: 
2025-07-17T11:07:26.0890901Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0891107Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0891318Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0891514Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0892012Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0892218Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0892432Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0892521Z 
2025-07-17T11:07:26.0892615Z SIGABRT(6), PID: 1800819, Thread 1800971: 
2025-07-17T11:07:26.0893088Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0893297Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0893507Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0893764Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0894262Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0894547Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0894762Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0894769Z 
2025-07-17T11:07:26.0894859Z SIGABRT(6), PID: 1800824, Thread 1801155: 
2025-07-17T11:07:26.0895332Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0895596Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0895808Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0896083Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0896585Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0896793Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0897010Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0897014Z 
2025-07-17T11:07:26.0897105Z SIGABRT(6), PID: 1800819, Thread 1800975: 
2025-07-17T11:07:26.0897574Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0897782Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0897995Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0898195Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0898695Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0898906Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0899120Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0899124Z 
2025-07-17T11:07:26.0899216Z SIGABRT(6), PID: 1800819, Thread 1800978: 
2025-07-17T11:07:26.0899689Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0899898Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0900110Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0900310Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0900812Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0901134Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0901358Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0901363Z 
2025-07-17T11:07:26.0901459Z SIGABRT(6), PID: 1800824, Thread 1801158: 
2025-07-17T11:07:26.0902002Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0902218Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0902421Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0902691Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0903186Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0903399Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0903678Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0903682Z 
2025-07-17T11:07:26.0903779Z SIGABRT(6), PID: 1800819, Thread 1800981: 
2025-07-17T11:07:26.0904249Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0904456Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0904666Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0904872Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0905372Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0905584Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0905796Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0905800Z 
2025-07-17T11:07:26.0905896Z SIGABRT(6), PID: 1800824, Thread 1801161: 
2025-07-17T11:07:26.0906364Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0906575Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0906783Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0906984Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0907479Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0907693Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0907901Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0907905Z 
2025-07-17T11:07:26.0908001Z SIGABRT(6), PID: 1800819, Thread 1800984: 
2025-07-17T11:07:26.0908468Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0908683Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0908891Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0909166Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0909661Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0909875Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0910140Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0910145Z 
2025-07-17T11:07:26.0910243Z SIGABRT(6), PID: 1800824, Thread 1801165: 
2025-07-17T11:07:26.0910714Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0910981Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0911190Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0911392Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0911885Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0912155Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0912363Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0912376Z 
2025-07-17T11:07:26.0912468Z SIGABRT(6), PID: 1800819, Thread 1800987: 
2025-07-17T11:07:26.0912949Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0913159Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0913372Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0913572Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0914075Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0914283Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0914495Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0914499Z 
2025-07-17T11:07:26.0914589Z SIGABRT(6), PID: 1800824, Thread 1801169: 
2025-07-17T11:07:26.0915059Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0915269Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0915478Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0915677Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0916179Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0916389Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0916598Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0916602Z 
2025-07-17T11:07:26.0916694Z SIGABRT(6), PID: 1800819, Thread 1800990: 
2025-07-17T11:07:26.0917166Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0917443Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0917653Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0917852Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0918406Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0918618Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0918833Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0919000Z 
2025-07-17T11:07:26.0919094Z SIGABRT(6), PID: 1800824, Thread 1801173: 
2025-07-17T11:07:26.0927340Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0927622Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0928024Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0928250Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0928799Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0929018Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0929242Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0929248Z 
2025-07-17T11:07:26.0929350Z SIGABRT(6), PID: 1800819, Thread 1800993: 
2025-07-17T11:07:26.0929854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0930073Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0930284Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0930490Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0931012Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0931222Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0931439Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0931446Z 
2025-07-17T11:07:26.0931545Z SIGABRT(6), PID: 1800824, Thread 1801177: 
2025-07-17T11:07:26.0932017Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0932228Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0932432Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0932633Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0933144Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0933354Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0933646Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0933651Z 
2025-07-17T11:07:26.0933747Z SIGABRT(6), PID: 1800819, Thread 1800997: 
2025-07-17T11:07:26.0934218Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0934432Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0934696Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0934902Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0935461Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0935681Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0935893Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0935897Z 
2025-07-17T11:07:26.0936082Z SIGABRT(6), PID: 1800819, Thread 1801000: 
2025-07-17T11:07:26.0936632Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0936847Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0937052Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0937255Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0937749Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0937963Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0938177Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0938184Z 
2025-07-17T11:07:26.0938279Z SIGABRT(6), PID: 1800824, Thread 1801181: 
2025-07-17T11:07:26.0938754Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0938964Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0939167Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0939365Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0939864Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0940077Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0940286Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0940292Z 
2025-07-17T11:07:26.0940386Z SIGABRT(6), PID: 1800819, Thread 1801003: 
2025-07-17T11:07:26.0940857Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0941069Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0941274Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0941475Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0942048Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0942261Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0942469Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0942562Z SIGABRT(6), PID: 1800824, Thread 1801185: 
2025-07-17T11:07:26.0943104Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0943313Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0943529Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0943804Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0944326Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0944537Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0944818Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0944823Z 
2025-07-17T11:07:26.0944826Z 
2025-07-17T11:07:26.0944920Z SIGABRT(6), PID: 1800824, Thread 1801189: 
2025-07-17T11:07:26.0945405Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0945614Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0945823Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0946023Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0946527Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0946733Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0946946Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0946950Z 
2025-07-17T11:07:26.0947040Z SIGABRT(6), PID: 1800819, Thread 1801007: 
2025-07-17T11:07:26.0947509Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0947717Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0947925Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0948126Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0948634Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0948843Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0949057Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0949061Z 
2025-07-17T11:07:26.0949151Z SIGABRT(6), PID: 1800819, Thread 1801011: 
2025-07-17T11:07:26.0949626Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0949832Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0950117Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0950316Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0950819Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0951083Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0951302Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0951306Z 
2025-07-17T11:07:26.0951396Z SIGABRT(6), PID: 1800824, Thread 1801193: 
2025-07-17T11:07:26.0951919Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0952132Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0952339Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0952596Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0953098Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0953307Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0953514Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0953518Z 
2025-07-17T11:07:26.0953611Z SIGABRT(6), PID: 1800819, Thread 1801015: 
2025-07-17T11:07:26.0954080Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0954297Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0954502Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0954706Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0955201Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0955410Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0955618Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0955623Z 
2025-07-17T11:07:26.0955716Z SIGABRT(6), PID: 1800824, Thread 1801197: 
2025-07-17T11:07:26.0956177Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0956390Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0956595Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0956794Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0957291Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0957498Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0957710Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0957714Z 
2025-07-17T11:07:26.0957809Z SIGABRT(6), PID: 1800819, Thread 1801020: 
2025-07-17T11:07:26.0958341Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0958550Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0958755Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0959013Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0959510Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0959721Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0960016Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0960023Z 
2025-07-17T11:07:26.0960118Z SIGABRT(6), PID: 1800824, Thread 1801201: 
2025-07-17T11:07:26.0960580Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0960852Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0961058Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0961259Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0961749Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0961959Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0962165Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0962169Z 
2025-07-17T11:07:26.0962261Z SIGABRT(6), PID: 1800819, Thread 1801024: 
2025-07-17T11:07:26.0962724Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0962935Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0963143Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0963349Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0963849Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0964057Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0964268Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0964272Z 
2025-07-17T11:07:26.0964361Z SIGABRT(6), PID: 1800824, Thread 1801205: 
2025-07-17T11:07:26.0964826Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0965032Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0965240Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0965435Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0965931Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0966206Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0966419Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0966425Z 
2025-07-17T11:07:26.0966515Z SIGABRT(6), PID: 1800819, Thread 1801028: 
2025-07-17T11:07:26.0966994Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0967261Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0967479Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0967681Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0968240Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0968454Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0968671Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0968730Z 
2025-07-17T11:07:26.0968823Z SIGABRT(6), PID: 1800824, Thread 1801209: 
2025-07-17T11:07:26.0969296Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0969498Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0969702Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0969899Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0970394Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0970600Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0970812Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0970816Z 
2025-07-17T11:07:26.0970906Z SIGABRT(6), PID: 1800819, Thread 1801032: 
2025-07-17T11:07:26.0971377Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0971584Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0971795Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0971992Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0972487Z frame #4: <unknown function> + 0x3475ab (0x7fa4b90875ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0972695Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0972908Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0972913Z 
2025-07-17T11:07:26.0973003Z SIGABRT(6), PID: 1800824, Thread 1801214: 
2025-07-17T11:07:26.0973472Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0973678Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0973886Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0974157Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0974652Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0974860Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0975066Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0975128Z 
2025-07-17T11:07:26.0975225Z SIGABRT(6), PID: 1800819, Thread 1801352: 
2025-07-17T11:07:26.0975689Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0976323Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fa3ca8e608a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0976542Z frame #2: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0976739Z frame #3: pthread_kill + 0x12c (0x7fa4bc1f19fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0976982Z frame #4: raise + 0x16 (0x7fa4bc19d476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0977146Z frame #5: abort + 0xd3 (0x7fa4bc1837f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0977388Z frame #6: <unknown function> + 0x972cc (0x7fa33846f2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0977626Z frame #7: <unknown function> + 0x95f47 (0x7fa33846df47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0977854Z frame #8: <unknown function> + 0x34b71 (0x7fa33840cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.0978072Z frame #9: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0978291Z frame #10: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0978297Z 
2025-07-17T11:07:26.0978394Z SIGABRT(6), PID: 1800824, Thread 1801218: 
2025-07-17T11:07:26.0978876Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0979093Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0979296Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0979500Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0980002Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0980215Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0980426Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0980430Z 
2025-07-17T11:07:26.0980525Z SIGABRT(6), PID: 1800819, Thread 1805076: 
2025-07-17T11:07:26.0980996Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0981208Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0981377Z frame #2: __poll + 0x4f (0x7fa4bc273bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0981788Z frame #3: <unknown function> + 0x489f (0x7fa4bbd7f89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.0982016Z frame #4: <unknown function> + 0x1483bc (0x555ec35233bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0982267Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0982574Z frame #6: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0982811Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0983033Z frame #8: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0983281Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x555ec350ba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0983597Z frame #10: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0983854Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x555ec3509c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0984083Z frame #12: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0984380Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0984608Z frame #14: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0984846Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0985127Z frame #16: <unknown function> + 0x147734 (0x555ec3522734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0985351Z frame #17: <unknown function> + 0x21ca82 (0x555ec35f7a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0985572Z frame #18: <unknown function> + 0x21c9e4 (0x555ec35f79e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.0985790Z frame #19: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0986002Z frame #20: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0986008Z 
2025-07-17T11:07:26.0986104Z SIGABRT(6), PID: 1800824, Thread 1801222: 
2025-07-17T11:07:26.0986587Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0986804Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0987011Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0987214Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0987717Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0987927Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0988136Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0988141Z 
2025-07-17T11:07:26.0988238Z SIGABRT(6), PID: 1800824, Thread 1801226: 
2025-07-17T11:07:26.0988708Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0988919Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0989120Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0989322Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0989821Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0990033Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0990241Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0990315Z 
2025-07-17T11:07:26.0990409Z SIGABRT(6), PID: 1800824, Thread 1801230: 
2025-07-17T11:07:26.0990887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0991096Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0991380Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0991578Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0992078Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0992339Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0992559Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0992564Z 
2025-07-17T11:07:26.0992656Z SIGABRT(6), PID: 1800819, Thread 1805077: 
2025-07-17T11:07:26.0993129Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0993398Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0993609Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0993833Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa4bc1ef2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0994287Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fa3f662e048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0994711Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fa3f6630bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.0994948Z frame #6: <unknown function> + 0xdc253 (0x7fa3c8b52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.0995158Z frame #7: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0995368Z frame #8: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0995372Z 
2025-07-17T11:07:26.0995466Z SIGABRT(6), PID: 1800824, Thread 1801234: 
2025-07-17T11:07:26.0995942Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0996147Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0996355Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0996554Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0997050Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0997256Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0997466Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0997470Z 
2025-07-17T11:07:26.0997558Z SIGABRT(6), PID: 1800824, Thread 1801239: 
2025-07-17T11:07:26.0998029Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.0998233Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0998439Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0998700Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0999205Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.0999412Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0999678Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.0999683Z 
2025-07-17T11:07:26.0999776Z SIGABRT(6), PID: 1800819, Thread 1805078: 
2025-07-17T11:07:26.1000311Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1000524Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1000736Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1000958Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa4bc1ef2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1001498Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fa3f6628741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1001731Z frame #5: <unknown function> + 0xdc253 (0x7fa3c8b52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1001944Z frame #6: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1002152Z frame #7: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1002160Z 
2025-07-17T11:07:26.1002251Z SIGABRT(6), PID: 1800824, Thread 1801242: 
2025-07-17T11:07:26.1002722Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1002927Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1003133Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1003332Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1003832Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1004037Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1004247Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1004254Z 
2025-07-17T11:07:26.1004344Z SIGABRT(6), PID: 1800819, Thread 1805081: 
2025-07-17T11:07:26.1004819Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1005026Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1005198Z frame #2: ioctl + 0x3f (0x7fa4bc27594f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1005433Z frame #3: <unknown function> + 0x13eef0 (0x7fa338516ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1005665Z frame #4: <unknown function> + 0x1378c4 (0x7fa33850f8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1005895Z frame #5: <unknown function> + 0x95b2b (0x7fa33846db2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1006124Z frame #6: <unknown function> + 0x34b71 (0x7fa33840cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1006334Z frame #7: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1006617Z frame #8: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1006622Z 
2025-07-17T11:07:26.1006712Z SIGABRT(6), PID: 1800824, Thread 1801245: 
2025-07-17T11:07:26.1007183Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1007392Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1007662Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1007863Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1008420Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1008629Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1008843Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1008847Z 
2025-07-17T11:07:26.1008936Z SIGABRT(6), PID: 1800819, Thread 1805094: 
2025-07-17T11:07:26.1009467Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1009674Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1009843Z frame #2: __poll + 0x4f (0x7fa4bc273bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1010054Z frame #3: <unknown function> + 0xd4540 (0x7fa380686540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1010267Z frame #4: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1010474Z frame #5: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1010480Z 
2025-07-17T11:07:26.1010573Z SIGABRT(6), PID: 1800819, Thread 1805102: 
2025-07-17T11:07:26.1011039Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1011256Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1011419Z frame #2: __poll + 0x4f (0x7fa4bc273bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1011636Z frame #3: <unknown function> + 0xd65a2 (0x7fa3806885a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1011843Z frame #4: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1012059Z frame #5: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1012063Z 
2025-07-17T11:07:26.1012152Z SIGABRT(6), PID: 1800824, Thread 1801248: 
2025-07-17T11:07:26.1012627Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1012834Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1013042Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1013243Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1013740Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1013943Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1014156Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1014228Z 
2025-07-17T11:07:26.1014321Z SIGABRT(6), PID: 1800819, Thread 1805113: 
2025-07-17T11:07:26.1014793Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1015007Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1015210Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1015470Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1015699Z frame #4: <unknown function> + 0xd1e7f (0x7fa380683e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1016008Z frame #5: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1016296Z frame #6: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1016304Z 
2025-07-17T11:07:26.1016403Z SIGABRT(6), PID: 1800824, Thread 1801252: 
2025-07-17T11:07:26.1016896Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1017183Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1017388Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1017596Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1018112Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1018331Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1018542Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1018552Z 
2025-07-17T11:07:26.1018654Z SIGABRT(6), PID: 1800819, Thread 1805117: 
2025-07-17T11:07:26.1019131Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1019350Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1019557Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1019771Z frame #3: <unknown function> + 0x9ca53 (0x7fa4bc1f7a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1020032Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x555ec34efbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1020268Z frame #5: <unknown function> + 0x1d314f (0x555ec35ae14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1020493Z frame #6: <unknown function> + 0x1d2f6b (0x555ec35adf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1020721Z frame #7: <unknown function> + 0x6df36 (0x555ec3448f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1020972Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1021203Z frame #9: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1021445Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1021680Z frame #11: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1021915Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1022145Z frame #13: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1022380Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1022714Z frame #15: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1022949Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1023178Z frame #17: <unknown function> + 0x147734 (0x555ec3522734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1023470Z frame #18: <unknown function> + 0x21ca82 (0x555ec35f7a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1023703Z frame #19: <unknown function> + 0x21c9e4 (0x555ec35f79e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1023920Z frame #20: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1024198Z frame #21: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1024203Z 
2025-07-17T11:07:26.1024299Z SIGABRT(6), PID: 1800824, Thread 1801255: 
2025-07-17T11:07:26.1024784Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1024997Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1025273Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1025478Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1025993Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1026202Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1026421Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1026426Z 
2025-07-17T11:07:26.1026517Z SIGABRT(6), PID: 1800820, Thread 1800820: 
2025-07-17T11:07:26.1027001Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1027213Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1027453Z frame #2: <unknown function> + 0x7741f (0x7f2a6df6341f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1027688Z frame #3: <unknown function> + 0x76c8e (0x7f2a6df62c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1027928Z frame #4: <unknown function> + 0x6ad69 (0x7f2a6df56d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1028149Z frame #5: <unknown function> + 0x3f54db (0x7f2afebbb4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1028371Z frame #6: <unknown function> + 0x3e11d8 (0x7f2afeba71d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1028587Z frame #7: <unknown function> + 0xb2f86 (0x7f2afe878f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1028799Z frame #8: <unknown function> + 0x8e501 (0x7f2afe854501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1029186Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f2b2965b7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.1029591Z frame #10: <unknown function> + 0xd307d9 (0x7f2b47bae7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1029817Z frame #11: <unknown function> + 0x139902 (0x55726c225902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1030081Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55726c21ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1030309Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1030555Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55726c21ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1030852Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1031098Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1031323Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1031625Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1031851Z frame #19: <unknown function> + 0x1474b2 (0x55726c2334b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1032060Z frame #20: PyObject_Call + 0xbc (0x55726c233e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1032340Z frame #21: <unknown function> + 0x1ff69d (0x55726c2eb69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1032573Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55726c221023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1032818Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55726c21ccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1033048Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1033345Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55726c2183bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1033576Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1033814Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55726c2183bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1034040Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1034281Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55726c21ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1034543Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55726c2204b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1034765Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55726c231ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1034989Z frame #32: <unknown function> + 0x205949 (0x55726c2f1949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1035212Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55726c221023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1035460Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55726c21ccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1035688Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1035928Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55726c2183bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1036153Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1036394Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55726c2183bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1036617Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1036859Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55726c2183bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1037081Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1037326Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55726c2193f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1037549Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1037948Z frame #44: <unknown function> + 0x7ab3f7 (0x7f2b476293f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1038173Z frame #45: <unknown function> + 0x13bc88 (0x55726c227c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1038383Z frame #46: PyObject_Call + 0x20f (0x55726c233f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1038698Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55726c21d98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1038922Z frame #48: <unknown function> + 0x147608 (0x55726c233608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1039164Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55726c21ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1039390Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1039683Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55726c2183bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1039916Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1040214Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55726c2193f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1040446Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1040685Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55726c2183bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1040912Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1041228Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55726c21ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1041454Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1041694Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55726c21ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1041916Z frame #60: <unknown function> + 0x147608 (0x55726c233608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1042155Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55726c21ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1042380Z frame #62: <unknown function> + 0x147608 (0x55726c233608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1042386Z 
2025-07-17T11:07:26.1042479Z SIGABRT(6), PID: 1800819, Thread 1805125: 
2025-07-17T11:07:26.1042975Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1043196Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1043371Z frame #2: read + 0x4c (0x7fa4bc26f81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1043556Z frame #3: _Py_read + 0x4c (0x555ec35b5a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1043787Z frame #4: <unknown function> + 0x208c3b (0x555ec35e3c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1044009Z frame #5: <unknown function> + 0x14648d (0x555ec352148d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1044265Z frame #6: PyObject_VectorcallMethod + 0x85 (0x555ec3530215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1044492Z frame #7: <unknown function> + 0x208ae7 (0x555ec35e3ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1044723Z frame #8: <unknown function> + 0x208472 (0x555ec35e3472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1044948Z frame #9: <unknown function> + 0x1da6ee (0x555ec35b56ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1045173Z frame #10: <unknown function> + 0x1483bc (0x555ec35233bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1045417Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1045648Z frame #12: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1045886Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x555ec35073bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1046110Z frame #14: <unknown function> + 0x147734 (0x555ec3522734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1046421Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x555ec3509c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1046651Z frame #16: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1046906Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1047135Z frame #18: _PyFunction_Vectorcall + 0x6c (0x555ec35170ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1047427Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x555ec35077b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1047659Z frame #20: <unknown function> + 0x147734 (0x555ec3522734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1047887Z frame #21: <unknown function> + 0x21ca82 (0x555ec35f7a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1048179Z frame #22: <unknown function> + 0x21c9e4 (0x555ec35f79e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1048400Z frame #23: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1048631Z frame #24: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1048635Z 
2025-07-17T11:07:26.1048811Z SIGABRT(6), PID: 1800824, Thread 1801258: 
2025-07-17T11:07:26.1049300Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1049519Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1049738Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1049939Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1050455Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1050666Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1050892Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1050898Z 
2025-07-17T11:07:26.1050991Z SIGABRT(6), PID: 1800820, Thread 1800912: 
2025-07-17T11:07:26.1051476Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1051691Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1051899Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1052104Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1052616Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1052837Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1053047Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1053051Z 
2025-07-17T11:07:26.1053151Z SIGABRT(6), PID: 1800824, Thread 1801262: 
2025-07-17T11:07:26.1053621Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1053835Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1054041Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1054253Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1054819Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1055030Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1055241Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1055247Z 
2025-07-17T11:07:26.1055344Z SIGABRT(6), PID: 1800820, Thread 1800918: 
2025-07-17T11:07:26.1056037Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1056262Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1056551Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1056769Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1057292Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1057585Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1057802Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1057806Z 
2025-07-17T11:07:26.1057907Z SIGABRT(6), PID: 1800824, Thread 1801265: 
2025-07-17T11:07:26.1058389Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1058607Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1058814Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1059020Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1059522Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1059735Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1059945Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1059950Z 
2025-07-17T11:07:26.1060046Z SIGABRT(6), PID: 1800820, Thread 1800922: 
2025-07-17T11:07:26.1060515Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1060733Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1060938Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1061138Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1061637Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1061853Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1062063Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1062159Z SIGABRT(6), PID: 1800824, Thread 1801267: 
2025-07-17T11:07:26.1062628Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1062842Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1063148Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1063350Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1063858Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1064136Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1064354Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1064358Z 
2025-07-17T11:07:26.1064361Z 
2025-07-17T11:07:26.1064450Z SIGABRT(6), PID: 1800824, Thread 1801271: 
2025-07-17T11:07:26.1064984Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1065196Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1065406Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1065665Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1066167Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1066374Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1066586Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1066590Z 
2025-07-17T11:07:26.1066679Z SIGABRT(6), PID: 1800820, Thread 1800926: 
2025-07-17T11:07:26.1067155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1067365Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1067571Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1067770Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1068265Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1068475Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1068691Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1068697Z 
2025-07-17T11:07:26.1068789Z SIGABRT(6), PID: 1800819, Thread 1805140: 
2025-07-17T11:07:26.1069263Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1069469Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1069680Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1069876Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1070262Z frame #4: <unknown function> + 0x5a02023 (0x7fa40061c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1070836Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa4006201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1071438Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa4006171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1071898Z frame #7: <unknown function> + 0x79d655 (0x7fa411b07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1072146Z frame #8: <unknown function> + 0xdc253 (0x7fa3c8b52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1072424Z frame #9: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1072655Z frame #10: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1072659Z 
2025-07-17T11:07:26.1072757Z SIGABRT(6), PID: 1800824, Thread 1801274: 
2025-07-17T11:07:26.1073288Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1073511Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1073718Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1073927Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1074493Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1074708Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1074921Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1074925Z 
2025-07-17T11:07:26.1075023Z SIGABRT(6), PID: 1800820, Thread 1800930: 
2025-07-17T11:07:26.1075504Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1075721Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1075929Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1076136Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1076635Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1076850Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1077061Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1077066Z 
2025-07-17T11:07:26.1077163Z SIGABRT(6), PID: 1800824, Thread 1801275: 
2025-07-17T11:07:26.1077634Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1077846Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1078052Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1078255Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1078755Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1078970Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1079176Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1079181Z 
2025-07-17T11:07:26.1079280Z SIGABRT(6), PID: 1800820, Thread 1800933: 
2025-07-17T11:07:26.1079819Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1080033Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1080239Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1080441Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1081001Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1081224Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1081513Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1081521Z 
2025-07-17T11:07:26.1081621Z SIGABRT(6), PID: 1800824, Thread 1801277: 
2025-07-17T11:07:26.1082091Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1082368Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1082572Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1082782Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1083295Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1083511Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1083719Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1083730Z 
2025-07-17T11:07:26.1083822Z SIGABRT(6), PID: 1800820, Thread 1800936: 
2025-07-17T11:07:26.1084306Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1084516Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1084726Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1084921Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1085427Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1085639Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1085861Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1085865Z 
2025-07-17T11:07:26.1085956Z SIGABRT(6), PID: 1800824, Thread 1801279: 
2025-07-17T11:07:26.1086432Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1086642Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1086854Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1087056Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1087555Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1087836Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1088051Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1088056Z 
2025-07-17T11:07:26.1088151Z SIGABRT(6), PID: 1800820, Thread 1800939: 
2025-07-17T11:07:26.1088639Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1088911Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1089135Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1089337Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1089908Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1090128Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1090347Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1090408Z 
2025-07-17T11:07:26.1090504Z SIGABRT(6), PID: 1800824, Thread 1801281: 
2025-07-17T11:07:26.1090990Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1091201Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1091410Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1091610Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1092114Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1092326Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1092545Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1092549Z 
2025-07-17T11:07:26.1092641Z SIGABRT(6), PID: 1800820, Thread 1800942: 
2025-07-17T11:07:26.1093122Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1093330Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1093541Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1093742Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1094245Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1094458Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1094668Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1094672Z 
2025-07-17T11:07:26.1094768Z SIGABRT(6), PID: 1800824, Thread 1801283: 
2025-07-17T11:07:26.1095236Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1095450Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1095659Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1095861Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1096516Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1096733Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1096942Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1096946Z 
2025-07-17T11:07:26.1097117Z SIGABRT(6), PID: 1800820, Thread 1800945: 
2025-07-17T11:07:26.1097611Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1097898Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1098131Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1098341Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1098853Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1099142Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1099357Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1099361Z 
2025-07-17T11:07:26.1099461Z SIGABRT(6), PID: 1800819, Thread 1805141: 
2025-07-17T11:07:26.1099944Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1100161Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1100371Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1100579Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1100959Z frame #4: <unknown function> + 0x5a02023 (0x7fa40061c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1101535Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa4006201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1102136Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa4006171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1102532Z frame #7: <unknown function> + 0x79d655 (0x7fa411b07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1102784Z frame #8: <unknown function> + 0xdc253 (0x7fa3c8b52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1103006Z frame #9: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1103227Z frame #10: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1103231Z 
2025-07-17T11:07:26.1103331Z SIGABRT(6), PID: 1800824, Thread 1801285: 
2025-07-17T11:07:26.1103806Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1104028Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1104243Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1104447Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1105041Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1105253Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1105472Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1105476Z 
2025-07-17T11:07:26.1105626Z SIGABRT(6), PID: 1800820, Thread 1800948: 
2025-07-17T11:07:26.1106110Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1106321Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1106591Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1106798Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1107311Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1107584Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1107800Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1107806Z 
2025-07-17T11:07:26.1107898Z SIGABRT(6), PID: 1800824, Thread 1801287: 
2025-07-17T11:07:26.1108378Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1108589Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1108801Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1109011Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1109521Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1109733Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1109950Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1109954Z 
2025-07-17T11:07:26.1110048Z SIGABRT(6), PID: 1800820, Thread 1800951: 
2025-07-17T11:07:26.1110533Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1110744Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1110960Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1111160Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1111669Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1111878Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1112093Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1112189Z SIGABRT(6), PID: 1800824, Thread 1801289: 
2025-07-17T11:07:26.1112665Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1112876Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1113166Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1113368Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1113876Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1114163Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1114384Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1114389Z 
2025-07-17T11:07:26.1114392Z 
2025-07-17T11:07:26.1114482Z SIGABRT(6), PID: 1800824, Thread 1801290: 
2025-07-17T11:07:26.1115014Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1115234Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1115440Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1115703Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1116200Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1116417Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1116626Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1116630Z 
2025-07-17T11:07:26.1116732Z SIGABRT(6), PID: 1800820, Thread 1800954: 
2025-07-17T11:07:26.1117205Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1117421Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1117630Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1117834Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1118337Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1118550Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1118767Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1118771Z 
2025-07-17T11:07:26.1118870Z SIGABRT(6), PID: 1800824, Thread 1801292: 
2025-07-17T11:07:26.1119341Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1119551Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1119758Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1119964Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1120461Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1120678Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1120887Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1120957Z 
2025-07-17T11:07:26.1121058Z SIGABRT(6), PID: 1800820, Thread 1800957: 
2025-07-17T11:07:26.1121526Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1121745Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1121955Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1122221Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1122725Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1122997Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1123213Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1123219Z 
2025-07-17T11:07:26.1123318Z SIGABRT(6), PID: 1800819, Thread 1805142: 
2025-07-17T11:07:26.1123792Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1124067Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1124277Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1124484Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1124862Z frame #4: <unknown function> + 0x5a02023 (0x7fa40061c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1125433Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa4006201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1126032Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa4006171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1126417Z frame #7: <unknown function> + 0x79d655 (0x7fa411b07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1126661Z frame #8: <unknown function> + 0xdc253 (0x7fa3c8b52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1126876Z frame #9: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1127098Z frame #10: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1127102Z 
2025-07-17T11:07:26.1127195Z SIGABRT(6), PID: 1800824, Thread 1801293: 
2025-07-17T11:07:26.1127672Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1127884Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1128101Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1128302Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1128813Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1129022Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1129244Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1129248Z 
2025-07-17T11:07:26.1129342Z SIGABRT(6), PID: 1800820, Thread 1800960: 
2025-07-17T11:07:26.1129898Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1130109Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1130327Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1130586Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1131099Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1131309Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1131583Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1131589Z 
2025-07-17T11:07:26.1131686Z SIGABRT(6), PID: 1800824, Thread 1801295: 
2025-07-17T11:07:26.1132168Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1132440Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1132654Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1132857Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1133359Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1133573Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1133795Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1133801Z 
2025-07-17T11:07:26.1133894Z SIGABRT(6), PID: 1800820, Thread 1800963: 
2025-07-17T11:07:26.1134379Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1134592Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1134810Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1135013Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1135523Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1135735Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1136033Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1136037Z 
2025-07-17T11:07:26.1136148Z SIGABRT(6), PID: 1800824, Thread 1801296: 
2025-07-17T11:07:26.1136621Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1136838Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1137044Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1137255Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1137757Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1138060Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1138275Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1138281Z 
2025-07-17T11:07:26.1138382Z SIGABRT(6), PID: 1800820, Thread 1800966: 
2025-07-17T11:07:26.1138854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1139137Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1139366Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1139586Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1140172Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1140399Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1140617Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1140710Z 
2025-07-17T11:07:26.1140811Z SIGABRT(6), PID: 1800824, Thread 1801298: 
2025-07-17T11:07:26.1141293Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1141511Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1141718Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1141929Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1142429Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1142648Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1142861Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1142865Z 
2025-07-17T11:07:26.1142966Z SIGABRT(6), PID: 1800820, Thread 1800969: 
2025-07-17T11:07:26.1143440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1143654Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1143863Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1144071Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1144572Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1144792Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1145002Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1145006Z 
2025-07-17T11:07:26.1145108Z SIGABRT(6), PID: 1800820, Thread 1800972: 
2025-07-17T11:07:26.1145579Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1145798Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1146009Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1146305Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1146810Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1147027Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1147238Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1147307Z 
2025-07-17T11:07:26.1147402Z SIGABRT(6), PID: 1800824, Thread 1801300: 
2025-07-17T11:07:26.1147881Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1148147Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1148368Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1148573Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1149079Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1149353Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1149581Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1149585Z 
2025-07-17T11:07:26.1149679Z SIGABRT(6), PID: 1800824, Thread 1801302: 
2025-07-17T11:07:26.1150156Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1150365Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1150581Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1150777Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1151282Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1151494Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1151711Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1151806Z SIGABRT(6), PID: 1800820, Thread 1800974: 
2025-07-17T11:07:26.1152289Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1152499Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1152711Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1152911Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1153418Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1153628Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1153848Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1153852Z 
2025-07-17T11:07:26.1153856Z 
2025-07-17T11:07:26.1153952Z SIGABRT(6), PID: 1800820, Thread 1800977: 
2025-07-17T11:07:26.1154429Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1154709Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1154923Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1155124Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1155681Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1155897Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1156115Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1156119Z 
2025-07-17T11:07:26.1156274Z SIGABRT(6), PID: 1800824, Thread 1801304: 
2025-07-17T11:07:26.1156757Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1156970Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1157241Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1157440Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1157951Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1158167Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1158377Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1158475Z SIGABRT(6), PID: 1800819, Thread 1805143: 
2025-07-17T11:07:26.1158962Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1159179Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1159390Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1159602Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1159982Z frame #4: <unknown function> + 0x5a02023 (0x7fa40061c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1160562Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa4006201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1161162Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa4006171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1161564Z frame #7: <unknown function> + 0x79d655 (0x7fa411b07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1161802Z frame #8: <unknown function> + 0xdc253 (0x7fa3c8b52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1162028Z frame #9: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1162247Z frame #10: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1162251Z 
2025-07-17T11:07:26.1162254Z 
2025-07-17T11:07:26.1162352Z SIGABRT(6), PID: 1800820, Thread 1800980: 
2025-07-17T11:07:26.1162829Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1163115Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1163324Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1163532Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1164042Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1164320Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1164536Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1164540Z 
2025-07-17T11:07:26.1164639Z SIGABRT(6), PID: 1800824, Thread 1801305: 
2025-07-17T11:07:26.1165169Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1165389Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1165600Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1165871Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1166373Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1166589Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1166800Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1166804Z 
2025-07-17T11:07:26.1166908Z SIGABRT(6), PID: 1800820, Thread 1800983: 
2025-07-17T11:07:26.1167383Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1167601Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1167808Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1168017Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1168517Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1168728Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1168946Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1168950Z 
2025-07-17T11:07:26.1169041Z SIGABRT(6), PID: 1800824, Thread 1801307: 
2025-07-17T11:07:26.1169521Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1169732Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1169947Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1170147Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1170647Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1170858Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1171076Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1171144Z 
2025-07-17T11:07:26.1171238Z SIGABRT(6), PID: 1800820, Thread 1800986: 
2025-07-17T11:07:26.1171714Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1171924Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1172136Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1172392Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1172900Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1173165Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1173388Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1173394Z 
2025-07-17T11:07:26.1173491Z SIGABRT(6), PID: 1800824, Thread 1801309: 
2025-07-17T11:07:26.1173970Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1174261Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1174478Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1174677Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1175185Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1175392Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1175611Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1175615Z 
2025-07-17T11:07:26.1175707Z SIGABRT(6), PID: 1800820, Thread 1800989: 
2025-07-17T11:07:26.1176255Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1176466Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1176677Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1176875Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1177387Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1177598Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1177815Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1177821Z 
2025-07-17T11:07:26.1177913Z SIGABRT(6), PID: 1800824, Thread 1801311: 
2025-07-17T11:07:26.1178390Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1178607Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1178815Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1179022Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1179513Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1179818Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1180027Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1180033Z 
2025-07-17T11:07:26.1180131Z SIGABRT(6), PID: 1800820, Thread 1800991: 
2025-07-17T11:07:26.1180669Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1180893Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1181101Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1181384Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1181902Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1182124Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1182407Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1182411Z 
2025-07-17T11:07:26.1182514Z SIGABRT(6), PID: 1800824, Thread 1801313: 
2025-07-17T11:07:26.1182999Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1183213Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1183420Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1183628Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1184127Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1184343Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1184554Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1184558Z 
2025-07-17T11:07:26.1184660Z SIGABRT(6), PID: 1800824, Thread 1801315: 
2025-07-17T11:07:26.1185127Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1185341Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1185550Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1185761Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1186257Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1186473Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1186688Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1186695Z 
2025-07-17T11:07:26.1186797Z SIGABRT(6), PID: 1800820, Thread 1800994: 
2025-07-17T11:07:26.1187263Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1187480Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1187685Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1188088Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1188592Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1188811Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1189087Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1189100Z 
2025-07-17T11:07:26.1189195Z SIGABRT(6), PID: 1800819, Thread 1805144: 
2025-07-17T11:07:26.1189674Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1189943Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1190166Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1190368Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1190816Z frame #4: <unknown function> + 0x5a02023 (0x7fa40061c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1191384Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa4006201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1191983Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa4006171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1192370Z frame #7: <unknown function> + 0x79d655 (0x7fa411b07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1192614Z frame #8: <unknown function> + 0xdc253 (0x7fa3c8b52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1192827Z frame #9: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1193058Z frame #10: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1193062Z 
2025-07-17T11:07:26.1193158Z SIGABRT(6), PID: 1800824, Thread 1801319: 
2025-07-17T11:07:26.1193645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1193857Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1194070Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1194272Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1194781Z frame #4: <unknown function> + 0x3475ab (0x7f3d110185ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1194993Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1195210Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1195215Z 
2025-07-17T11:07:26.1195311Z SIGABRT(6), PID: 1800820, Thread 1800998: 
2025-07-17T11:07:26.1195791Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1196000Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1196213Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1196481Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1196988Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1197201Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1197476Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1197482Z 
2025-07-17T11:07:26.1197577Z SIGABRT(6), PID: 1800824, Thread 1801338: 
2025-07-17T11:07:26.1198059Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1198571Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f3c2287708a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1198799Z frame #2: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1198998Z frame #3: pthread_kill + 0x12c (0x7f3d141829fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1199239Z frame #4: raise + 0x16 (0x7f3d1412e476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1199403Z frame #5: abort + 0xd3 (0x7f3d141147f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1199662Z frame #6: <unknown function> + 0x972cc (0x7f3b904002cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1199899Z frame #7: <unknown function> + 0x95f47 (0x7f3b903fef47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1200134Z frame #8: <unknown function> + 0x34b71 (0x7f3b9039db71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1200351Z frame #9: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1200576Z frame #10: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1200582Z 
2025-07-17T11:07:26.1200676Z SIGABRT(6), PID: 1800820, Thread 1801002: 
2025-07-17T11:07:26.1201161Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1201381Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1201589Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1201792Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1202306Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1202527Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1202742Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1202746Z 
2025-07-17T11:07:26.1202847Z SIGABRT(6), PID: 1800824, Thread 1803864: 
2025-07-17T11:07:26.1203319Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1203539Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1203711Z frame #2: __poll + 0x4f (0x7f3d14204bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1204132Z frame #3: <unknown function> + 0x489f (0x7f3d13d1089f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.1204363Z frame #4: <unknown function> + 0x1483bc (0x5590349723bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1204627Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1204929Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1205172Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1205398Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1205715Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55903495aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1205952Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1206213Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x559034958c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1206525Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1206774Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1207001Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1207244Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1207527Z frame #16: <unknown function> + 0x147734 (0x559034971734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1207756Z frame #17: <unknown function> + 0x21ca82 (0x559034a46a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1207979Z frame #18: <unknown function> + 0x21c9e4 (0x559034a469e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1208203Z frame #19: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1208415Z frame #20: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1208421Z 
2025-07-17T11:07:26.1208520Z SIGABRT(6), PID: 1800820, Thread 1801006: 
2025-07-17T11:07:26.1208996Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1209224Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1209436Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1209650Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1210157Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1210374Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1210584Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1210590Z 
2025-07-17T11:07:26.1210691Z SIGABRT(6), PID: 1800820, Thread 1801010: 
2025-07-17T11:07:26.1211161Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1211378Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1211583Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1211789Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1212288Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1212508Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1212717Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1212786Z 
2025-07-17T11:07:26.1212893Z SIGABRT(6), PID: 1800820, Thread 1801014: 
2025-07-17T11:07:26.1213364Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1213580Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1213843Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1214057Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1214550Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1214829Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1215048Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1215052Z 
2025-07-17T11:07:26.1215155Z SIGABRT(6), PID: 1800824, Thread 1803865: 
2025-07-17T11:07:26.1215629Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1215975Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1216184Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1216419Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f3d141802dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1216870Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f3c4e5bf048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1217304Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f3c4e5c1bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1217548Z frame #6: <unknown function> + 0xdc253 (0x7f3c20ae3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1217761Z frame #7: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1217983Z frame #8: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1217988Z 
2025-07-17T11:07:26.1218082Z SIGABRT(6), PID: 1800819, Thread 1805145: 
2025-07-17T11:07:26.1218558Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1218769Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1218984Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1219185Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1219568Z frame #4: <unknown function> + 0x5a02023 (0x7fa40061c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1220137Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa4006201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1220737Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa4006171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1221120Z frame #7: <unknown function> + 0x79d655 (0x7fa411b07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1221359Z frame #8: <unknown function> + 0xdc253 (0x7fa3c8b52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1221658Z frame #9: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1221885Z frame #10: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1221893Z 
2025-07-17T11:07:26.1221991Z SIGABRT(6), PID: 1800820, Thread 1801018: 
2025-07-17T11:07:26.1222553Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1222770Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1222992Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1223280Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1223801Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1224016Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1224303Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1224308Z 
2025-07-17T11:07:26.1224401Z SIGABRT(6), PID: 1800820, Thread 1801022: 
2025-07-17T11:07:26.1224887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1225099Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1225316Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1225518Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1226027Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1226242Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1226467Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1226471Z 
2025-07-17T11:07:26.1226569Z SIGABRT(6), PID: 1800824, Thread 1803866: 
2025-07-17T11:07:26.1227053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1227270Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1227479Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1227716Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f3d141802dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1228199Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f3c4e5b9741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1228440Z frame #5: <unknown function> + 0xdc253 (0x7f3c20ae3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1228651Z frame #6: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1228868Z frame #7: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1228872Z 
2025-07-17T11:07:26.1228967Z SIGABRT(6), PID: 1800824, Thread 1803867: 
2025-07-17T11:07:26.1229449Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1229661Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1229911Z frame #2: ioctl + 0x3f (0x7f3d1420694f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1230154Z frame #3: <unknown function> + 0x13eef0 (0x7f3b904a7ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1230396Z frame #4: <unknown function> + 0x1378c4 (0x7f3b904a08c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1230633Z frame #5: <unknown function> + 0x95b2b (0x7f3b903feb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1230929Z frame #6: <unknown function> + 0x34b71 (0x7f3b9039db71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1231144Z frame #7: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1231363Z frame #8: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1231367Z 
2025-07-17T11:07:26.1231519Z SIGABRT(6), PID: 1800820, Thread 1801026: 
2025-07-17T11:07:26.1232007Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1232220Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1232493Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1232699Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1233217Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1233430Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1233648Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1233654Z 
2025-07-17T11:07:26.1233748Z SIGABRT(6), PID: 1800824, Thread 1805099: 
2025-07-17T11:07:26.1234233Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1234441Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1234624Z frame #2: __poll + 0x4f (0x7f3d14204bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1234840Z frame #3: <unknown function> + 0xd4540 (0x7f3bd8617540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1235054Z frame #4: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1235262Z frame #5: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1235266Z 
2025-07-17T11:07:26.1235362Z SIGABRT(6), PID: 1800820, Thread 1801030: 
2025-07-17T11:07:26.1235835Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1236050Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1236255Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1236462Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1236961Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1237179Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1237391Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1237396Z 
2025-07-17T11:07:26.1237494Z SIGABRT(6), PID: 1800824, Thread 1805105: 
2025-07-17T11:07:26.1238029Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1238240Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1238408Z frame #2: __poll + 0x4f (0x7f3d14204bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1238625Z frame #3: <unknown function> + 0xd65a2 (0x7f3bd86195a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1238919Z frame #4: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1239138Z frame #5: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1239142Z 
2025-07-17T11:07:26.1239235Z SIGABRT(6), PID: 1800820, Thread 1801034: 
2025-07-17T11:07:26.1239767Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1239980Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1240192Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1240450Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1240965Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1241184Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1241396Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1241489Z SIGABRT(6), PID: 1800824, Thread 1805115: 
2025-07-17T11:07:26.1241970Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1242188Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1242391Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1242601Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1242817Z frame #4: <unknown function> + 0xd1e7f (0x7f3bd8614e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1243033Z frame #5: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1243240Z frame #6: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1243244Z 
2025-07-17T11:07:26.1243248Z 
2025-07-17T11:07:26.1243348Z SIGABRT(6), PID: 1800820, Thread 1801038: 
2025-07-17T11:07:26.1243821Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1244038Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1244244Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1244453Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1244956Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1245172Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1245385Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1245390Z 
2025-07-17T11:07:26.1245493Z SIGABRT(6), PID: 1800824, Thread 1805120: 
2025-07-17T11:07:26.1246035Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1246248Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1246457Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1246671Z frame #3: <unknown function> + 0x9ca53 (0x7f3d14188a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1246984Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55903493ebc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1247220Z frame #5: <unknown function> + 0x1d314f (0x5590349fd14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1247443Z frame #6: <unknown function> + 0x1d2f6b (0x5590349fcf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1247731Z frame #7: <unknown function> + 0x6df36 (0x559034897f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1247976Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1248206Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1248506Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1248741Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1248990Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1249220Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1249454Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1249686Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1249926Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1250149Z frame #17: <unknown function> + 0x147734 (0x559034971734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1250369Z frame #18: <unknown function> + 0x21ca82 (0x559034a46a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1250599Z frame #19: <unknown function> + 0x21c9e4 (0x559034a469e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1250813Z frame #20: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1251035Z frame #21: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1251039Z 
2025-07-17T11:07:26.1251133Z SIGABRT(6), PID: 1800819, Thread 1805146: 
2025-07-17T11:07:26.1251617Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1251837Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1252054Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1252264Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1252657Z frame #4: <unknown function> + 0x5a02023 (0x7fa40061c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1253231Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa4006201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1253833Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa4006171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1254282Z frame #7: <unknown function> + 0x79d655 (0x7fa411b07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1254520Z frame #8: <unknown function> + 0xdc253 (0x7fa3c8b52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1254737Z frame #9: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1254961Z frame #10: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1255114Z SIGABRT(6), PID: 1800820, Thread 1801043: 
2025-07-17T11:07:26.1255602Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1255874Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1256183Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1256393Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1256909Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1257201Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1257425Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1257429Z 
2025-07-17T11:07:26.1257433Z 
2025-07-17T11:07:26.1257531Z SIGABRT(6), PID: 1800824, Thread 1805133: 
2025-07-17T11:07:26.1258002Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1258219Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1258392Z frame #2: read + 0x4c (0x7f3d1420081c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1258583Z frame #3: _Py_read + 0x4c (0x559034a04a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1258809Z frame #4: <unknown function> + 0x208c3b (0x559034a32c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1259037Z frame #5: <unknown function> + 0x14648d (0x55903497048d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1259283Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55903497f215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1259510Z frame #7: <unknown function> + 0x208ae7 (0x559034a32ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1259730Z frame #8: <unknown function> + 0x208472 (0x559034a32472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1259960Z frame #9: <unknown function> + 0x1da6ee (0x559034a046ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1260183Z frame #10: <unknown function> + 0x1483bc (0x5590349723bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1260429Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1260659Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1260903Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5590349563bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1261123Z frame #14: <unknown function> + 0x147734 (0x559034971734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1261376Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x559034958c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1261601Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1261843Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1262066Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5590349660ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1262384Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5590349567b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1262603Z frame #20: <unknown function> + 0x147734 (0x559034971734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1262829Z frame #21: <unknown function> + 0x21ca82 (0x559034a46a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1263123Z frame #22: <unknown function> + 0x21c9e4 (0x559034a469e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1263356Z frame #23: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1263571Z frame #24: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1263575Z 
2025-07-17T11:07:26.1263675Z SIGABRT(6), PID: 1800820, Thread 1801047: 
2025-07-17T11:07:26.1264219Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1264446Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1264659Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1264932Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1265451Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1265671Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1265881Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1265885Z 
2025-07-17T11:07:26.1265987Z SIGABRT(6), PID: 1800820, Thread 1801050: 
2025-07-17T11:07:26.1266463Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1266681Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1266887Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1267099Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1267605Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1267822Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1268036Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1268040Z 
2025-07-17T11:07:26.1268141Z SIGABRT(6), PID: 1800823, Thread 1800823: 
2025-07-17T11:07:26.1268612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1268828Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1269065Z frame #2: <unknown function> + 0x7741f (0x7f390f83c41f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1269305Z frame #3: <unknown function> + 0x76c8e (0x7f390f83bc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1269542Z frame #4: <unknown function> + 0x6ad69 (0x7f390f82fd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1269767Z frame #5: <unknown function> + 0x3f54db (0x7f39a04944db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1269980Z frame #6: <unknown function> + 0x3e11d8 (0x7f39a04801d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1270205Z frame #7: <unknown function> + 0xb2f86 (0x7f39a0151f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1270511Z frame #8: <unknown function> + 0x8e501 (0x7f39a012d501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1270906Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f39caf347ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.1271300Z frame #10: <unknown function> + 0xd307d9 (0x7f39e94877d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1271596Z frame #11: <unknown function> + 0x139902 (0x556da325d902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1271853Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x556da3254a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1272094Z frame #13: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1272404Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x556da3254a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1272647Z frame #15: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1272890Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1273183Z frame #17: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1273423Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1273655Z frame #19: <unknown function> + 0x1474b2 (0x556da326b4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1273865Z frame #20: PyObject_Call + 0xbc (0x556da326be4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1274095Z frame #21: <unknown function> + 0x1ff69d (0x556da332369d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1274326Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x556da3259023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1274585Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x556da3254cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1274812Z frame #24: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1275064Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x556da32503bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1275288Z frame #26: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1275537Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x556da32503bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1275760Z frame #28: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1276012Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x556da3254a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1276271Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x556da32584b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1276499Z frame #31: _PyObject_Call_Prepend + 0x69 (0x556da3269ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1276721Z frame #32: <unknown function> + 0x205949 (0x556da3329949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1276951Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x556da3259023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1277198Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x556da3254cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1277436Z frame #35: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1277671Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x556da32503bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1277903Z frame #37: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1278141Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x556da32503bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1278442Z frame #39: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1278677Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x556da32503bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1278919Z frame #41: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1279164Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x556da32513f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1279451Z frame #43: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1279847Z frame #44: <unknown function> + 0x7ab3f7 (0x7f39e8f023f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1280079Z frame #45: <unknown function> + 0x13bc88 (0x556da325fc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1280343Z frame #46: PyObject_Call + 0x20f (0x556da326bf9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1280606Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x556da325598d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1280830Z frame #48: <unknown function> + 0x147608 (0x556da326b608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1281137Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x556da3252c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1281365Z frame #50: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1281613Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x556da32503bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1281836Z frame #52: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1282083Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x556da32513f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1282311Z frame #54: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1282555Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x556da32503bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1282780Z frame #56: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1283031Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x556da3252c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1283259Z frame #58: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1283504Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x556da3252c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1283723Z frame #60: <unknown function> + 0x147608 (0x556da326b608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1283966Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x556da3252c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1284188Z frame #62: <unknown function> + 0x147608 (0x556da326b608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1284196Z 
2025-07-17T11:07:26.1284298Z SIGABRT(6), PID: 1800820, Thread 1801054: 
2025-07-17T11:07:26.1284780Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1285004Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1285212Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1285428Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1285935Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1286154Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1286368Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1286442Z 
2025-07-17T11:07:26.1286544Z SIGABRT(6), PID: 1800823, Thread 1801036: 
2025-07-17T11:07:26.1287021Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1287235Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1287501Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1287717Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1288219Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1288493Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1288714Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1288722Z 
2025-07-17T11:07:26.1288815Z SIGABRT(6), PID: 1800820, Thread 1801058: 
2025-07-17T11:07:26.1289298Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1289567Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1289782Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1289982Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1290491Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1290700Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1290921Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1290925Z 
2025-07-17T11:07:26.1291017Z SIGABRT(6), PID: 1800823, Thread 1801040: 
2025-07-17T11:07:26.1291498Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1291705Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1291916Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1292121Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1292628Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1292841Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1293054Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1293060Z 
2025-07-17T11:07:26.1293153Z SIGABRT(6), PID: 1800820, Thread 1801062: 
2025-07-17T11:07:26.1293640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1293851Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1294067Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1294270Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1300861Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1301288Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1301540Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1301546Z 
2025-07-17T11:07:26.1301650Z SIGABRT(6), PID: 1800819, Thread 1805147: 
2025-07-17T11:07:26.1302262Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa3ca8e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1302501Z frame #1: <unknown function> + 0x42520 (0x7fa4bc19d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1302720Z frame #2: <unknown function> + 0x91117 (0x7fa4bc1ec117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1303013Z frame #3: pthread_cond_wait + 0x211 (0x7fa4bc1eea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1303415Z frame #4: <unknown function> + 0x5a02023 (0x7fa40061c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1303995Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa4006201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1304704Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa4006171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1305098Z frame #7: <unknown function> + 0x79d655 (0x7fa411b07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1305342Z frame #8: <unknown function> + 0xdc253 (0x7fa3c8b52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1305558Z frame #9: <unknown function> + 0x94ac3 (0x7fa4bc1efac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1305785Z frame #10: <unknown function> + 0x126850 (0x7fa4bc281850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1305790Z 
2025-07-17T11:07:26.1305890Z SIGABRT(6), PID: 1800823, Thread 1801045: 
2025-07-17T11:07:26.1306379Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1306606Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1306823Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1307041Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1307562Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1307785Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1308002Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1308006Z 
2025-07-17T11:07:26.1308109Z SIGABRT(6), PID: 1800820, Thread 1801066: 
2025-07-17T11:07:26.1308584Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1308807Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1309016Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1309224Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1309744Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1310033Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1310246Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1310252Z 
2025-07-17T11:07:26.1310349Z SIGABRT(6), PID: 1800824, Thread 1805164: 
2025-07-17T11:07:26.1310902Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1311120Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1311325Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1311587Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1311970Z frame #4: <unknown function> + 0x5a02023 (0x7f3c585ad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1312548Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3c585b11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1313206Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3c585a81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1313602Z frame #7: <unknown function> + 0x79d655 (0x7f3c69a98655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1313837Z frame #8: <unknown function> + 0xdc253 (0x7f3c20ae3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1314053Z frame #9: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1314273Z frame #10: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1314277Z 
2025-07-17T11:07:26.1314372Z SIGABRT(6), PID: 1800822, Thread 1800822: 
2025-07-17T11:07:26.1314842Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1315057Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1315294Z frame #2: <unknown function> + 0x7741f (0x7f215121841f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1315528Z frame #3: <unknown function> + 0x76c8e (0x7f2151217c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1315750Z frame #4: <unknown function> + 0x6ad69 (0x7f215120bd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1315974Z frame #5: <unknown function> + 0x3f54db (0x7f21e1e704db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1316188Z frame #6: <unknown function> + 0x3e11d8 (0x7f21e1e5c1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1316406Z frame #7: <unknown function> + 0xb2f86 (0x7f21e1b2df86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1316612Z frame #8: <unknown function> + 0x8e501 (0x7f21e1b09501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1316999Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f220c9107ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.1317384Z frame #10: <unknown function> + 0xd307d9 (0x7f222ae637d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1317617Z frame #11: <unknown function> + 0x139902 (0x562c7760b902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1317877Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x562c77602a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1318108Z frame #13: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1318426Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x562c77602a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1318657Z frame #15: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1318900Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1319133Z frame #17: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1319431Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1319661Z frame #19: <unknown function> + 0x1474b2 (0x562c776194b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1319866Z frame #20: PyObject_Call + 0xbc (0x562c77619e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1320162Z frame #21: <unknown function> + 0x1ff69d (0x562c776d169d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1320395Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x562c77607023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1320645Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x562c77602cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1321040Z frame #24: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1321283Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x562c775fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1321510Z frame #26: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1321754Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x562c775fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1321977Z frame #28: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1322225Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x562c77602a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1322486Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x562c776064b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1322709Z frame #31: _PyObject_Call_Prepend + 0x69 (0x562c77617ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1322929Z frame #32: <unknown function> + 0x205949 (0x562c776d7949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1323162Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x562c77607023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1323403Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x562c77602cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1323632Z frame #35: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1323870Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x562c775fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1324103Z frame #37: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1324339Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x562c775fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1324569Z frame #39: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1324806Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x562c775fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1325031Z frame #41: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1325276Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x562c775ff3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1325505Z frame #43: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1325896Z frame #44: <unknown function> + 0x7ab3f7 (0x7f222a8de3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1326130Z frame #45: <unknown function> + 0x13bc88 (0x562c7760dc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1326411Z frame #46: PyObject_Call + 0x20f (0x562c77619f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1326661Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x562c7760398d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1326884Z frame #48: <unknown function> + 0x147608 (0x562c77619608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1327134Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x562c77600c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1327427Z frame #50: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1327672Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x562c775fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1327897Z frame #52: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1328196Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x562c775ff3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1328424Z frame #54: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1328666Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x562c775fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1328952Z frame #56: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1329204Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x562c77600c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1329431Z frame #58: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1329671Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x562c77600c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1329886Z frame #60: <unknown function> + 0x147608 (0x562c77619608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1330129Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x562c77600c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1330344Z frame #62: <unknown function> + 0x147608 (0x562c77619608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1330351Z 
2025-07-17T11:07:26.1330449Z SIGABRT(6), PID: 1800820, Thread 1801070: 
2025-07-17T11:07:26.1330928Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1331149Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1331364Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1331569Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1332086Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1332298Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1332512Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1332605Z SIGABRT(6), PID: 1800823, Thread 1801048: 
2025-07-17T11:07:26.1333083Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1333289Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1333501Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1333700Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1334207Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1334487Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1334703Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1334711Z 
2025-07-17T11:07:26.1334714Z 
2025-07-17T11:07:26.1334805Z SIGABRT(6), PID: 1800820, Thread 1801073: 
2025-07-17T11:07:26.1335336Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1335547Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1335760Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1336120Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1336633Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1336843Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1337132Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1337137Z 
2025-07-17T11:07:26.1337230Z SIGABRT(6), PID: 1800822, Thread 1800995: 
2025-07-17T11:07:26.1337705Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1337910Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1338119Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1338320Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1338824Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1339034Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1339251Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1339256Z 
2025-07-17T11:07:26.1339346Z SIGABRT(6), PID: 1800823, Thread 1801052: 
2025-07-17T11:07:26.1339821Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1340027Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1340238Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1340436Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1340931Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1341141Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1341356Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1341361Z 
2025-07-17T11:07:26.1341453Z SIGABRT(6), PID: 1800818, Thread 1800818: 
2025-07-17T11:07:26.1341921Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1342133Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1342369Z frame #2: <unknown function> + 0x77445 (0x7fa0f3c2f445 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1342690Z frame #3: <unknown function> + 0x76c8e (0x7fa0f3c2ec8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1342920Z frame #4: <unknown function> + 0x6ad69 (0x7fa0f3c22d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1343144Z frame #5: <unknown function> + 0x3f54db (0x7fa1848874db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1343358Z frame #6: <unknown function> + 0x3e11d8 (0x7fa1848731d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1343644Z frame #7: <unknown function> + 0xb2f86 (0x7fa184544f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1343855Z frame #8: <unknown function> + 0x8e501 (0x7fa184520501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.1344297Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fa1af3277ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.1344688Z frame #10: <unknown function> + 0xd307d9 (0x7fa1cd8767d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1344925Z frame #11: <unknown function> + 0x139902 (0x5644aecba902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1345172Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5644aecb1a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1345470Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1345712Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5644aecb1a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1345943Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1346181Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1346412Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1346648Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1346875Z frame #19: <unknown function> + 0x1474b2 (0x5644aecc84b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1347080Z frame #20: PyObject_Call + 0xbc (0x5644aecc8e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1347307Z frame #21: <unknown function> + 0x1ff69d (0x5644aed8069d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1347532Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5644aecb6023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1347779Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5644aecb1cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1348004Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1348247Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5644aecad3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1348472Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1348717Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5644aecad3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1348941Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1349186Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5644aecb1a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1349441Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5644aecb54b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1349676Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5644aecc6ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1349899Z frame #32: <unknown function> + 0x205949 (0x5644aed86949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1350130Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5644aecb6023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1350373Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5644aecb1cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1350683Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1350919Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5644aecad3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1351148Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1351383Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5644aecad3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1351666Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1351904Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5644aecad3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1352131Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1352430Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5644aecae3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1352657Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1353056Z frame #44: <unknown function> + 0x7ab3f7 (0x7fa1cd2f13f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1353343Z frame #45: <unknown function> + 0x13bc88 (0x5644aecbcc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1353553Z frame #46: PyObject_Call + 0x20f (0x5644aecc8f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1353799Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5644aecb298d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1354021Z frame #48: <unknown function> + 0x147608 (0x5644aecc8608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1354265Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5644aecafc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1354491Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1354732Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5644aecad3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1354956Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1355198Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5644aecae3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1355420Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1355656Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5644aecad3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1355883Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1356126Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5644aecafc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1356356Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1356595Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5644aecafc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1356815Z frame #60: <unknown function> + 0x147608 (0x5644aecc8608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1357055Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5644aecafc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1357277Z frame #62: <unknown function> + 0x147608 (0x5644aecc8608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1357285Z 
2025-07-17T11:07:26.1357383Z SIGABRT(6), PID: 1800820, Thread 1801077: 
2025-07-17T11:07:26.1357853Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1358070Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1358278Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1358553Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1359063Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1359281Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1359549Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1359553Z 
2025-07-17T11:07:26.1359652Z SIGABRT(6), PID: 1800822, Thread 1800999: 
2025-07-17T11:07:26.1360122Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1360391Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1360604Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1360809Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1361367Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1361583Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1361794Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1361798Z 
2025-07-17T11:07:26.1361892Z SIGABRT(6), PID: 1800823, Thread 1801056: 
2025-07-17T11:07:26.1362361Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1362572Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1362774Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1362975Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1363473Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1363686Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1363897Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1363902Z 
2025-07-17T11:07:26.1363998Z SIGABRT(6), PID: 1800820, Thread 1801081: 
2025-07-17T11:07:26.1364465Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1364678Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1364880Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1365085Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1365583Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1365797Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1366004Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1366009Z 
2025-07-17T11:07:26.1366104Z SIGABRT(6), PID: 1800818, Thread 1800826: 
2025-07-17T11:07:26.1366574Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1366857Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1367064Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1367267Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1367837Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1368054Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1368323Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1368328Z 
2025-07-17T11:07:26.1368422Z SIGABRT(6), PID: 1800822, Thread 1801004: 
2025-07-17T11:07:26.1368900Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1369108Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1369379Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1369579Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1370079Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1370288Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1370502Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1370508Z 
2025-07-17T11:07:26.1370597Z SIGABRT(6), PID: 1800823, Thread 1801060: 
2025-07-17T11:07:26.1371070Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1371276Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1371483Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1371682Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1372181Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1372389Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1372604Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1372610Z 
2025-07-17T11:07:26.1372701Z SIGABRT(6), PID: 1800820, Thread 1801085: 
2025-07-17T11:07:26.1373168Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1373375Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1373589Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1373787Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1374286Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1374496Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1374776Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1374781Z 
2025-07-17T11:07:26.1374871Z SIGABRT(6), PID: 1800823, Thread 1801064: 
2025-07-17T11:07:26.1375341Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1375548Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1375813Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1376080Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1376652Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1376867Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1377081Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1377085Z 
2025-07-17T11:07:26.1377251Z SIGABRT(6), PID: 1800822, Thread 1801009: 
2025-07-17T11:07:26.1377732Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1377941Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1378149Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1378347Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1378849Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1379061Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1379267Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1379359Z SIGABRT(6), PID: 1800818, Thread 1800827: 
2025-07-17T11:07:26.1379832Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1380046Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1380251Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1380449Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1380945Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1381163Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1381370Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1381376Z 
2025-07-17T11:07:26.1381380Z 
2025-07-17T11:07:26.1381473Z SIGABRT(6), PID: 1800820, Thread 1801089: 
2025-07-17T11:07:26.1381945Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1382155Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1382361Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1382563Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1383147Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1383363Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1383573Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1383577Z 
2025-07-17T11:07:26.1383672Z SIGABRT(6), PID: 1800823, Thread 1801068: 
2025-07-17T11:07:26.1384207Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1384420Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1384683Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1384887Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1385381Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1385658Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1385864Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1385868Z 
2025-07-17T11:07:26.1385963Z SIGABRT(6), PID: 1800822, Thread 1801013: 
2025-07-17T11:07:26.1386428Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1386638Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1386840Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1387046Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1387534Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1387746Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1387956Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1387960Z 
2025-07-17T11:07:26.1388054Z SIGABRT(6), PID: 1800824, Thread 1805165: 
2025-07-17T11:07:26.1388519Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1388731Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1388939Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1389136Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1389517Z frame #4: <unknown function> + 0x5a02023 (0x7f3c585ad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1390096Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3c585b11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1390698Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3c585a81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1391082Z frame #7: <unknown function> + 0x79d655 (0x7f3c69a98655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1391421Z frame #8: <unknown function> + 0xdc253 (0x7f3c20ae3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1391629Z frame #9: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1391853Z frame #10: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1391857Z 
2025-07-17T11:07:26.1391949Z SIGABRT(6), PID: 1800818, Thread 1800828: 
2025-07-17T11:07:26.1392486Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1392700Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1392974Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1393180Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1393693Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1393900Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1394183Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1394187Z 
2025-07-17T11:07:26.1394280Z SIGABRT(6), PID: 1800820, Thread 1801093: 
2025-07-17T11:07:26.1394755Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1394962Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1395175Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1395375Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1395876Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1396086Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1396303Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1396308Z 
2025-07-17T11:07:26.1396398Z SIGABRT(6), PID: 1800823, Thread 1801071: 
2025-07-17T11:07:26.1396870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1397083Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1397295Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1397494Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1397996Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1398206Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1398425Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1398429Z 
2025-07-17T11:07:26.1398519Z SIGABRT(6), PID: 1800822, Thread 1801016: 
2025-07-17T11:07:26.1398993Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1399215Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1399518Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1399722Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1400235Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1400505Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1400721Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1400726Z 
2025-07-17T11:07:26.1400823Z SIGABRT(6), PID: 1800818, Thread 1800829: 
2025-07-17T11:07:26.1401349Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1401570Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1401777Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1401984Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1402538Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1402750Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1402962Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1402967Z 
2025-07-17T11:07:26.1403063Z SIGABRT(6), PID: 1800820, Thread 1801096: 
2025-07-17T11:07:26.1403532Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1403746Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1403951Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1404160Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1404660Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1404881Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1405092Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1405190Z SIGABRT(6), PID: 1800823, Thread 1801074: 
2025-07-17T11:07:26.1405655Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1405871Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1406077Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1406281Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1406774Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1406992Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1407199Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1407300Z SIGABRT(6), PID: 1800822, Thread 1801019: 
2025-07-17T11:07:26.1407772Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1408053Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1408262Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1408463Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1409014Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1409238Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1409507Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1409610Z SIGABRT(6), PID: 1800818, Thread 1800830: 
2025-07-17T11:07:26.1410088Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1410302Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1410566Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1410771Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1411269Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1411486Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1411696Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1411709Z 
2025-07-17T11:07:26.1411713Z 
2025-07-17T11:07:26.1411716Z 
2025-07-17T11:07:26.1411719Z 
2025-07-17T11:07:26.1411811Z SIGABRT(6), PID: 1800822, Thread 1801023: 
2025-07-17T11:07:26.1412285Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1412497Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1412715Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1412917Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1413416Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1413625Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1413843Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1413848Z 
2025-07-17T11:07:26.1413939Z SIGABRT(6), PID: 1800820, Thread 1801100: 
2025-07-17T11:07:26.1414414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1414624Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1414842Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1415044Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1415550Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1415827Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1416112Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1416117Z 
2025-07-17T11:07:26.1416210Z SIGABRT(6), PID: 1800818, Thread 1800831: 
2025-07-17T11:07:26.1416688Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1416975Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1417192Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1417395Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1417969Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1418184Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1418404Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1418571Z SIGABRT(6), PID: 1800823, Thread 1801078: 
2025-07-17T11:07:26.1419059Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1419282Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1419501Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1419701Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1420214Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1420423Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1420643Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1420649Z 
2025-07-17T11:07:26.1420654Z 
2025-07-17T11:07:26.1420751Z SIGABRT(6), PID: 1800822, Thread 1801027: 
2025-07-17T11:07:26.1421232Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1421446Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1421649Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1421853Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1422353Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1422567Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1422776Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1422780Z 
2025-07-17T11:07:26.1422878Z SIGABRT(6), PID: 1800820, Thread 1801104: 
2025-07-17T11:07:26.1423350Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1423567Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1423772Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1423976Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1424560Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1424784Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1424997Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1425001Z 
2025-07-17T11:07:26.1425157Z SIGABRT(6), PID: 1800824, Thread 1805166: 
2025-07-17T11:07:26.1425630Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1425899Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1426108Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1426319Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1426698Z frame #4: <unknown function> + 0x5a02023 (0x7f3c585ad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1427362Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3c585b11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1427960Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3c585a81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1428353Z frame #7: <unknown function> + 0x79d655 (0x7f3c69a98655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1428592Z frame #8: <unknown function> + 0xdc253 (0x7f3c20ae3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1428808Z frame #9: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1429024Z frame #10: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1429029Z 
2025-07-17T11:07:26.1429130Z SIGABRT(6), PID: 1800818, Thread 1800832: 
2025-07-17T11:07:26.1429605Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1429830Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1430035Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1430246Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1430743Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1430961Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1431175Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1431186Z 
2025-07-17T11:07:26.1431277Z SIGABRT(6), PID: 1800823, Thread 1801082: 
2025-07-17T11:07:26.1431757Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1431965Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1432180Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1432383Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1432970Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1433184Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1433402Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1433406Z 
2025-07-17T11:07:26.1433553Z SIGABRT(6), PID: 1800822, Thread 1801031: 
2025-07-17T11:07:26.1434036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1434242Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1434512Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1434717Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1435224Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1435492Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1435708Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1435715Z 
2025-07-17T11:07:26.1435807Z SIGABRT(6), PID: 1800820, Thread 1801109: 
2025-07-17T11:07:26.1436282Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1436491Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1436702Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1436905Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1437414Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1437626Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1437846Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1437850Z 
2025-07-17T11:07:26.1437939Z SIGABRT(6), PID: 1800823, Thread 1801086: 
2025-07-17T11:07:26.1438414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1438621Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1438835Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1439033Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1439545Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1439754Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1439976Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1439980Z 
2025-07-17T11:07:26.1440072Z SIGABRT(6), PID: 1800818, Thread 1800833: 
2025-07-17T11:07:26.1440552Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1440831Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1441052Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1441254Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1441764Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1442043Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1442257Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1442261Z 
2025-07-17T11:07:26.1442359Z SIGABRT(6), PID: 1800822, Thread 1801035: 
2025-07-17T11:07:26.1442891Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1443120Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1443326Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1443597Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1444097Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1444315Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1444525Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1444529Z 
2025-07-17T11:07:26.1444625Z SIGABRT(6), PID: 1800820, Thread 1801113: 
2025-07-17T11:07:26.1445094Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1445316Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1445521Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1445723Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1446224Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1446440Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1446651Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1446655Z 
2025-07-17T11:07:26.1446752Z SIGABRT(6), PID: 1800823, Thread 1801090: 
2025-07-17T11:07:26.1447221Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1447433Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1447638Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1447844Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1448341Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1448558Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1448769Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1448846Z 
2025-07-17T11:07:26.1448950Z SIGABRT(6), PID: 1800822, Thread 1801039: 
2025-07-17T11:07:26.1449421Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1449635Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1449850Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1450115Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1450612Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1451010Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1451224Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1451231Z 
2025-07-17T11:07:26.1451329Z SIGABRT(6), PID: 1800818, Thread 1800834: 
2025-07-17T11:07:26.1451801Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1452080Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1452291Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1452499Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1453001Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1453215Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1453433Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1453436Z 
2025-07-17T11:07:26.1453528Z SIGABRT(6), PID: 1800820, Thread 1801117: 
2025-07-17T11:07:26.1454010Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1454220Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1454432Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1454631Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1455135Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1455344Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1455559Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1455563Z 
2025-07-17T11:07:26.1455653Z SIGABRT(6), PID: 1800823, Thread 1801095: 
2025-07-17T11:07:26.1456196Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1456405Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1456616Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1456815Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1457318Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1457612Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1457831Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1457837Z 
2025-07-17T11:07:26.1457928Z SIGABRT(6), PID: 1800822, Thread 1801042: 
2025-07-17T11:07:26.1458475Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1458687Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1458903Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1459199Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1459718Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1459931Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1460280Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1460284Z 
2025-07-17T11:07:26.1460375Z SIGABRT(6), PID: 1800818, Thread 1800835: 
2025-07-17T11:07:26.1460854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1461062Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1461279Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1461478Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1461983Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1462197Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1462420Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1462423Z 
2025-07-17T11:07:26.1462515Z SIGABRT(6), PID: 1800820, Thread 1801121: 
2025-07-17T11:07:26.1462997Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1463211Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1463420Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1463629Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1464130Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1464350Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1464564Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1464568Z 
2025-07-17T11:07:26.1464667Z SIGABRT(6), PID: 1800823, Thread 1801099: 
2025-07-17T11:07:26.1465137Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1465378Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1465600Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1465887Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1466387Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1466604Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1466877Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1466884Z 
2025-07-17T11:07:26.1466984Z SIGABRT(6), PID: 1800818, Thread 1800836: 
2025-07-17T11:07:26.1467458Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1467733Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1467950Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1468157Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1468660Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1468965Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1469176Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1469275Z SIGABRT(6), PID: 1800822, Thread 1801046: 
2025-07-17T11:07:26.1469745Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1469965Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1470173Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1470376Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1470876Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1471093Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1471305Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1471309Z 
2025-07-17T11:07:26.1471313Z 
2025-07-17T11:07:26.1471412Z SIGABRT(6), PID: 1800820, Thread 1801124: 
2025-07-17T11:07:26.1471883Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1472099Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1472303Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1472513Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1473012Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1473234Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1473447Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1473457Z 
2025-07-17T11:07:26.1473553Z SIGABRT(6), PID: 1800823, Thread 1801103: 
2025-07-17T11:07:26.1474036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1474324Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1474534Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1474740Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1475302Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1475519Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1475738Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1475805Z 
2025-07-17T11:07:26.1475899Z SIGABRT(6), PID: 1800818, Thread 1800837: 
2025-07-17T11:07:26.1476392Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1476605Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1476881Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1477083Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1477585Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1477796Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1478014Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1478018Z 
2025-07-17T11:07:26.1478111Z SIGABRT(6), PID: 1800822, Thread 1801051: 
2025-07-17T11:07:26.1478585Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1478798Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1479009Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1479214Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1479730Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1479942Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1480161Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1480166Z 
2025-07-17T11:07:26.1480257Z SIGABRT(6), PID: 1800820, Thread 1801129: 
2025-07-17T11:07:26.1480729Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1480945Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1481162Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1481363Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1481881Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1482097Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1482390Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1482394Z 
2025-07-17T11:07:26.1482488Z SIGABRT(6), PID: 1800823, Thread 1801107: 
2025-07-17T11:07:26.1482963Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1483172Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1483443Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1483649Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1484204Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1484420Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1484646Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1484650Z 
2025-07-17T11:07:26.1484741Z SIGABRT(6), PID: 1800824, Thread 1805167: 
2025-07-17T11:07:26.1485277Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1485497Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1485704Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1485920Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1486301Z frame #4: <unknown function> + 0x5a02023 (0x7f3c585ad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1486887Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3c585b11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1487490Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3c585a81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1487884Z frame #7: <unknown function> + 0x79d655 (0x7f3c69a98655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1488121Z frame #8: <unknown function> + 0xdc253 (0x7f3c20ae3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1488335Z frame #9: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1488555Z frame #10: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1488562Z 
2025-07-17T11:07:26.1488661Z SIGABRT(6), PID: 1800818, Thread 1800838: 
2025-07-17T11:07:26.1489130Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1489353Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1489562Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1489775Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1490274Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1490495Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1490709Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1490780Z 
2025-07-17T11:07:26.1490879Z SIGABRT(6), PID: 1800822, Thread 1801055: 
2025-07-17T11:07:26.1491351Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1491565Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1491847Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1492062Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1492561Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1492837Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1493054Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1493058Z 
2025-07-17T11:07:26.1493156Z SIGABRT(6), PID: 1800820, Thread 1801134: 
2025-07-17T11:07:26.1493682Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1493897Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1494106Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1494314Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1494826Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1495045Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1495256Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1495359Z SIGABRT(6), PID: 1800823, Thread 1801111: 
2025-07-17T11:07:26.1495841Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1496118Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1496330Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1496528Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1497035Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1497255Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1497473Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1497479Z 
2025-07-17T11:07:26.1497482Z 
2025-07-17T11:07:26.1497573Z SIGABRT(6), PID: 1800818, Thread 1800839: 
2025-07-17T11:07:26.1498051Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1498256Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1498468Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1498670Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1499171Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1499471Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1499692Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1499696Z 
2025-07-17T11:07:26.1499791Z SIGABRT(6), PID: 1800822, Thread 1801059: 
2025-07-17T11:07:26.1500344Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1500557Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1500766Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1501035Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1501542Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1501751Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1502048Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1502052Z 
2025-07-17T11:07:26.1502147Z SIGABRT(6), PID: 1800820, Thread 1801139: 
2025-07-17T11:07:26.1502630Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1502841Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1503057Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1503259Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1503759Z frame #4: <unknown function> + 0x3475ab (0x7f2beeb9b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1503969Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1504185Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1504190Z 
2025-07-17T11:07:26.1504285Z SIGABRT(6), PID: 1800823, Thread 1801115: 
2025-07-17T11:07:26.1504788Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1505011Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1505227Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1505432Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1505947Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1506167Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1506381Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1506385Z 
2025-07-17T11:07:26.1506483Z SIGABRT(6), PID: 1800820, Thread 1801342: 
2025-07-17T11:07:26.1506962Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1507445Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f2b003fa08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1507740Z frame #2: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1507947Z frame #3: pthread_kill + 0x12c (0x7f2bf1d059fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1508121Z frame #4: raise + 0x16 (0x7f2bf1cb1476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1508290Z frame #5: abort + 0xd3 (0x7f2bf1c977f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1508587Z frame #6: <unknown function> + 0x972cc (0x7f2a6df832cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1508830Z frame #7: <unknown function> + 0x95f47 (0x7f2a6df81f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1509061Z frame #8: <unknown function> + 0x34b71 (0x7f2a6df20b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1509341Z frame #9: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1509566Z frame #10: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1509667Z SIGABRT(6), PID: 1800818, Thread 1800840: 
2025-07-17T11:07:26.1510158Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1510438Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1510647Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1510856Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1511366Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1511580Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1511792Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1511796Z 
2025-07-17T11:07:26.1511800Z 
2025-07-17T11:07:26.1511902Z SIGABRT(6), PID: 1800822, Thread 1801063: 
2025-07-17T11:07:26.1512396Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1512615Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1512821Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1513033Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1513534Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1513752Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1513960Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1513965Z 
2025-07-17T11:07:26.1514064Z SIGABRT(6), PID: 1800823, Thread 1801119: 
2025-07-17T11:07:26.1514537Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1514752Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1514960Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1515170Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1515670Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1515957Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1516167Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1516173Z 
2025-07-17T11:07:26.1516272Z SIGABRT(6), PID: 1800818, Thread 1800841: 
2025-07-17T11:07:26.1516804Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1517022Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1517228Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1517491Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1517995Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1518210Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1518478Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1518489Z 
2025-07-17T11:07:26.1518584Z SIGABRT(6), PID: 1800820, Thread 1804672: 
2025-07-17T11:07:26.1519069Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1519275Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1519461Z frame #2: __poll + 0x4f (0x7f2bf1d87bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1519879Z frame #3: <unknown function> + 0x489f (0x7f2bf189389f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.1520130Z frame #4: <unknown function> + 0x1483bc (0x55726c2343bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1520381Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1520616Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1520858Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1521092Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1521342Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55726c21ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1521584Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1521836Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55726c21ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1522071Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1522309Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1522545Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1522787Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1523023Z frame #16: <unknown function> + 0x147734 (0x55726c233734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1523249Z frame #17: <unknown function> + 0x21ca82 (0x55726c308a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1523477Z frame #18: <unknown function> + 0x21c9e4 (0x55726c3089e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1523693Z frame #19: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1524011Z frame #20: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1524015Z 
2025-07-17T11:07:26.1524111Z SIGABRT(6), PID: 1800822, Thread 1801067: 
2025-07-17T11:07:26.1524604Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1524886Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1525140Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1525347Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1525915Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1526134Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1526349Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1526411Z 
2025-07-17T11:07:26.1526508Z SIGABRT(6), PID: 1800823, Thread 1801123: 
2025-07-17T11:07:26.1526987Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1527195Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1527408Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1527611Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1528119Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1528334Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1528551Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1528556Z 
2025-07-17T11:07:26.1528654Z SIGABRT(6), PID: 1800818, Thread 1800842: 
2025-07-17T11:07:26.1529133Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1529341Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1529550Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1529753Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1530254Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1530466Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1530683Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1530687Z 
2025-07-17T11:07:26.1530779Z SIGABRT(6), PID: 1800822, Thread 1801072: 
2025-07-17T11:07:26.1531258Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1531472Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1531684Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1531883Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1532456Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1532669Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1532896Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1532900Z 
2025-07-17T11:07:26.1533060Z SIGABRT(6), PID: 1800823, Thread 1801127: 
2025-07-17T11:07:26.1533538Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1533750Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1534016Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1534224Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1534722Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1534995Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1535208Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1535214Z 
2025-07-17T11:07:26.1535311Z SIGABRT(6), PID: 1800818, Thread 1800843: 
2025-07-17T11:07:26.1535786Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1536064Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1536269Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1536483Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1536989Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1537210Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1537420Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1537424Z 
2025-07-17T11:07:26.1537519Z SIGABRT(6), PID: 1800822, Thread 1801076: 
2025-07-17T11:07:26.1538012Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1538229Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1538435Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1538639Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1539148Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1539362Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1539573Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1539578Z 
2025-07-17T11:07:26.1539675Z SIGABRT(6), PID: 1800824, Thread 1805168: 
2025-07-17T11:07:26.1540148Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1540455Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1540667Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1540882Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1541269Z frame #4: <unknown function> + 0x5a02023 (0x7f3c585ad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1541924Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3c585b11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1542599Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3c585a81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1543007Z frame #7: <unknown function> + 0x79d655 (0x7f3c69a98655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1543256Z frame #8: <unknown function> + 0xdc253 (0x7f3c20ae3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1543548Z frame #9: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1543773Z frame #10: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1543777Z 
2025-07-17T11:07:26.1543873Z SIGABRT(6), PID: 1800823, Thread 1801131: 
2025-07-17T11:07:26.1544365Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1544576Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1544791Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1544998Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1545509Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1545730Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1545946Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1545950Z 
2025-07-17T11:07:26.1546041Z SIGABRT(6), PID: 1800822, Thread 1801079: 
2025-07-17T11:07:26.1546522Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1546733Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1546947Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1547149Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1547651Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1547870Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1548091Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1548095Z 
2025-07-17T11:07:26.1548186Z SIGABRT(6), PID: 1800818, Thread 1800844: 
2025-07-17T11:07:26.1548675Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1548885Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1549171Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1549372Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1549877Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1550147Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1550404Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1550408Z 
2025-07-17T11:07:26.1550504Z SIGABRT(6), PID: 1800820, Thread 1804673: 
2025-07-17T11:07:26.1551057Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1551273Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1551489Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1551799Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f2bf1d032dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1552258Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f2b2c142048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1552689Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f2b2c144bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1552940Z frame #6: <unknown function> + 0xdc253 (0x7f2afe666253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1553157Z frame #7: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1553378Z frame #8: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1553384Z 
2025-07-17T11:07:26.1553480Z SIGABRT(6), PID: 1800823, Thread 1801137: 
2025-07-17T11:07:26.1553962Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1554180Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1554387Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1554595Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1555098Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1555330Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1555544Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1555548Z 
2025-07-17T11:07:26.1555651Z SIGABRT(6), PID: 1800822, Thread 1801083: 
2025-07-17T11:07:26.1556125Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1556341Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1556549Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1556756Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1557257Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1557547Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1557759Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1557763Z 
2025-07-17T11:07:26.1557864Z SIGABRT(6), PID: 1800818, Thread 1800845: 
2025-07-17T11:07:26.1558344Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1558623Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1558833Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1559037Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1559592Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1559816Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1560034Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1560196Z SIGABRT(6), PID: 1800823, Thread 1801142: 
2025-07-17T11:07:26.1560675Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1560891Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1561099Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1561306Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1561802Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1562025Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1562237Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1562241Z 
2025-07-17T11:07:26.1562244Z 
2025-07-17T11:07:26.1562344Z SIGABRT(6), PID: 1800822, Thread 1801087: 
2025-07-17T11:07:26.1562843Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1563059Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1563263Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1563474Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1563975Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1564191Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1564401Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1564410Z 
2025-07-17T11:07:26.1564502Z SIGABRT(6), PID: 1800820, Thread 1804674: 
2025-07-17T11:07:26.1564985Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1565192Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1565412Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1565719Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f2bf1d032dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1566200Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f2b2c13c741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1566436Z frame #5: <unknown function> + 0xdc253 (0x7f2afe666253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1566650Z frame #6: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1566918Z frame #7: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1566923Z 
2025-07-17T11:07:26.1567022Z SIGABRT(6), PID: 1800820, Thread 1804675: 
2025-07-17T11:07:26.1567562Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1567779Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1567963Z frame #2: ioctl + 0x3f (0x7f2bf1d8994f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1568210Z frame #3: <unknown function> + 0x13eef0 (0x7f2a6e02aef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1568505Z frame #4: <unknown function> + 0x1378c4 (0x7f2a6e0238c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1568746Z frame #5: <unknown function> + 0x95b2b (0x7f2a6df81b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1568980Z frame #6: <unknown function> + 0x34b71 (0x7f2a6df20b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1569193Z frame #7: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1569407Z frame #8: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1569413Z 
2025-07-17T11:07:26.1569517Z SIGABRT(6), PID: 1800823, Thread 1801146: 
2025-07-17T11:07:26.1569991Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1570203Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1570415Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1570629Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1571135Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1571357Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1571568Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1571667Z SIGABRT(6), PID: 1800822, Thread 1801091: 
2025-07-17T11:07:26.1572139Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1572355Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1572562Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1572777Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1573276Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1573495Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1573705Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1573776Z 
2025-07-17T11:07:26.1573878Z SIGABRT(6), PID: 1800818, Thread 1800846: 
2025-07-17T11:07:26.1574349Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1574567Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1574775Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1575045Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1575562Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1575839Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1576127Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1576133Z 
2025-07-17T11:07:26.1576136Z 
2025-07-17T11:07:26.1576236Z SIGABRT(6), PID: 1800823, Thread 1801150: 
2025-07-17T11:07:26.1576715Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1577028Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1577255Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1577458Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1577979Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1578197Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1578415Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1578418Z 
2025-07-17T11:07:26.1578510Z SIGABRT(6), PID: 1800822, Thread 1801094: 
2025-07-17T11:07:26.1578987Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1579195Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1579405Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1579605Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1580108Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1580320Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1580537Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1580543Z 
2025-07-17T11:07:26.1580635Z SIGABRT(6), PID: 1800820, Thread 1805093: 
2025-07-17T11:07:26.1581115Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1581324Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1581497Z frame #2: __poll + 0x4f (0x7f2bf1d87bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1581715Z frame #3: <unknown function> + 0xd4540 (0x7f2ab619a540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1581931Z frame #4: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1582227Z frame #5: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1582232Z 
2025-07-17T11:07:26.1582331Z SIGABRT(6), PID: 1800818, Thread 1800847: 
2025-07-17T11:07:26.1582826Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1583045Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1583433Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1583642Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1584231Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1584449Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1584663Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1584667Z 
2025-07-17T11:07:26.1584765Z SIGABRT(6), PID: 1800823, Thread 1801153: 
2025-07-17T11:07:26.1585304Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1585523Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1585725Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1585932Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1586431Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1586648Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1586858Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1586871Z 
2025-07-17T11:07:26.1586963Z SIGABRT(6), PID: 1800820, Thread 1805101: 
2025-07-17T11:07:26.1587440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1587652Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1587845Z frame #2: __poll + 0x4f (0x7f2bf1d87bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1588087Z frame #3: <unknown function> + 0xd65a2 (0x7f2ab619c5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1588303Z frame #4: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1588514Z frame #5: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1588518Z 
2025-07-17T11:07:26.1588613Z SIGABRT(6), PID: 1800822, Thread 1801098: 
2025-07-17T11:07:26.1589083Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1589295Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1589501Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1589706Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1590213Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1590499Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1590713Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1590717Z 
2025-07-17T11:07:26.1590814Z SIGABRT(6), PID: 1800820, Thread 1805110: 
2025-07-17T11:07:26.1591283Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1591554Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1591762Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1591968Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1592240Z frame #4: <unknown function> + 0xd1e7f (0x7f2ab6197e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1592457Z frame #5: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1592675Z frame #6: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1592679Z 
2025-07-17T11:07:26.1592774Z SIGABRT(6), PID: 1800818, Thread 1800848: 
2025-07-17T11:07:26.1593309Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1593525Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1593731Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1593937Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1594440Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1594655Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1594865Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1594870Z 
2025-07-17T11:07:26.1594967Z SIGABRT(6), PID: 1800823, Thread 1801156: 
2025-07-17T11:07:26.1595453Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1595666Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1595870Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1596078Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1596578Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1596795Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1597004Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1597007Z 
2025-07-17T11:07:26.1597105Z SIGABRT(6), PID: 1800822, Thread 1801102: 
2025-07-17T11:07:26.1597576Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1597792Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1597998Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1598204Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1598778Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1598996Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1599211Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1599220Z 
2025-07-17T11:07:26.1599311Z SIGABRT(6), PID: 1800824, Thread 1805169: 
2025-07-17T11:07:26.1599843Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1600060Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1600333Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1600540Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1600930Z frame #4: <unknown function> + 0x5a02023 (0x7f3c585ad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1601504Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3c585b11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1602171Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3c585a81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1602574Z frame #7: <unknown function> + 0x79d655 (0x7f3c69a98655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1602818Z frame #8: <unknown function> + 0xdc253 (0x7f3c20ae3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1603034Z frame #9: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1603257Z frame #10: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1603261Z 
2025-07-17T11:07:26.1603353Z SIGABRT(6), PID: 1800818, Thread 1800849: 
2025-07-17T11:07:26.1603837Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1604052Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1604262Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1604464Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1604974Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1605188Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1605404Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1605495Z SIGABRT(6), PID: 1800820, Thread 1805118: 
2025-07-17T11:07:26.1605972Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1606178Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1606391Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1606600Z frame #3: <unknown function> + 0x9ca53 (0x7f2bf1d0ba53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1606864Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55726c200bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1607166Z frame #5: <unknown function> + 0x1d314f (0x55726c2bf14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1607397Z frame #6: <unknown function> + 0x1d2f6b (0x55726c2bef6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1607622Z frame #7: <unknown function> + 0x6df36 (0x55726c159f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1607876Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1608160Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1608413Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1608641Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1608936Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1609164Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1609406Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1609695Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1609954Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1610180Z frame #17: <unknown function> + 0x147734 (0x55726c233734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1610406Z frame #18: <unknown function> + 0x21ca82 (0x55726c308a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1610626Z frame #19: <unknown function> + 0x21c9e4 (0x55726c3089e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1610849Z frame #20: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1611063Z frame #21: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1611067Z 
2025-07-17T11:07:26.1611071Z 
2025-07-17T11:07:26.1611169Z SIGABRT(6), PID: 1800823, Thread 1801159: 
2025-07-17T11:07:26.1611647Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1611869Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1612075Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1612282Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1612789Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1613005Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1613217Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1613226Z 
2025-07-17T11:07:26.1613318Z SIGABRT(6), PID: 1800822, Thread 1801106: 
2025-07-17T11:07:26.1613800Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1614009Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1614224Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1614425Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1614946Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1615229Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1615445Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1615451Z 
2025-07-17T11:07:26.1615543Z SIGABRT(6), PID: 1800818, Thread 1800850: 
2025-07-17T11:07:26.1616223Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1616452Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1616671Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1616949Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1617473Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1617691Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1617987Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1617991Z 
2025-07-17T11:07:26.1618085Z SIGABRT(6), PID: 1800823, Thread 1801162: 
2025-07-17T11:07:26.1618581Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1618792Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1619003Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1619207Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1619719Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1619934Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1620158Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1620162Z 
2025-07-17T11:07:26.1620254Z SIGABRT(6), PID: 1800820, Thread 1805129: 
2025-07-17T11:07:26.1620739Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1620949Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1621124Z frame #2: read + 0x4c (0x7f2bf1d8381c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1621310Z frame #3: _Py_read + 0x4c (0x55726c2c6a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1621545Z frame #4: <unknown function> + 0x208c3b (0x55726c2f4c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1621769Z frame #5: <unknown function> + 0x14648d (0x55726c23248d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1622022Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55726c241215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1622262Z frame #7: <unknown function> + 0x208ae7 (0x55726c2f4ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1622486Z frame #8: <unknown function> + 0x208472 (0x55726c2f4472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1622710Z frame #9: <unknown function> + 0x1da6ee (0x55726c2c66ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1622942Z frame #10: <unknown function> + 0x1483bc (0x55726c2343bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1623188Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1623505Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1623744Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55726c2183bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1623972Z frame #14: <unknown function> + 0x147734 (0x55726c233734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1624227Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55726c21ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1624517Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1624754Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1624987Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55726c2280ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1625281Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55726c2187b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1625512Z frame #20: <unknown function> + 0x147734 (0x55726c233734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1625734Z frame #21: <unknown function> + 0x21ca82 (0x55726c308a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1626014Z frame #22: <unknown function> + 0x21c9e4 (0x55726c3089e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1626229Z frame #23: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1626451Z frame #24: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1626455Z 
2025-07-17T11:07:26.1626548Z SIGABRT(6), PID: 1800822, Thread 1801110: 
2025-07-17T11:07:26.1627035Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1627250Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1627471Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1627676Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1628190Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1628401Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1628614Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1628617Z 
2025-07-17T11:07:26.1628708Z SIGABRT(6), PID: 1800818, Thread 1800851: 
2025-07-17T11:07:26.1629190Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1629398Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1629611Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1629833Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1630345Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1630556Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1630770Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1630774Z 
2025-07-17T11:07:26.1630870Z SIGABRT(6), PID: 1800823, Thread 1801166: 
2025-07-17T11:07:26.1631341Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1631630Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1631836Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1632044Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1632632Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1632855Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1633064Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1633068Z 
2025-07-17T11:07:26.1633221Z SIGABRT(6), PID: 1800822, Thread 1801114: 
2025-07-17T11:07:26.1633697Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1633913Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1634181Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1634387Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1634904Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1635132Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1635343Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1635347Z 
2025-07-17T11:07:26.1635442Z SIGABRT(6), PID: 1800818, Thread 1800852: 
2025-07-17T11:07:26.1635917Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1636130Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1636338Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1636545Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1637040Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1637255Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1637469Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1637475Z 
2025-07-17T11:07:26.1637575Z SIGABRT(6), PID: 1800823, Thread 1801170: 
2025-07-17T11:07:26.1638048Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1638261Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1638464Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1638671Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1639169Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1639386Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1639594Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1639667Z 
2025-07-17T11:07:26.1639767Z SIGABRT(6), PID: 1800822, Thread 1801118: 
2025-07-17T11:07:26.1640246Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1640461Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1640729Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1640934Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1641490Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1641709Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1641928Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1641932Z 
2025-07-17T11:07:26.1642023Z SIGABRT(6), PID: 1800818, Thread 1800853: 
2025-07-17T11:07:26.1642580Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1642794Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1643006Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1643203Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1643714Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1643925Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1644140Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1644145Z 
2025-07-17T11:07:26.1644236Z SIGABRT(6), PID: 1800823, Thread 1801174: 
2025-07-17T11:07:26.1644720Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1644932Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1645143Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1645344Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1645841Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1646049Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1646262Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1646268Z 
2025-07-17T11:07:26.1646357Z SIGABRT(6), PID: 1800822, Thread 1801122: 
2025-07-17T11:07:26.1646836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1647041Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1647251Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1647467Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1647969Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1648270Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1648494Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1648498Z 
2025-07-17T11:07:26.1648587Z SIGABRT(6), PID: 1800818, Thread 1800854: 
2025-07-17T11:07:26.1649122Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1649334Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1649603Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1649805Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1650315Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1650525Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1650800Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1650804Z 
2025-07-17T11:07:26.1650897Z SIGABRT(6), PID: 1800823, Thread 1801178: 
2025-07-17T11:07:26.1651381Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1651594Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1651801Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1652004Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1652515Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1652739Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1652952Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1652957Z 
2025-07-17T11:07:26.1653052Z SIGABRT(6), PID: 1800822, Thread 1801126: 
2025-07-17T11:07:26.1653523Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1653735Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1653940Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1654147Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1654640Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1654855Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1655068Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1655072Z 
2025-07-17T11:07:26.1655177Z SIGABRT(6), PID: 1800818, Thread 1800855: 
2025-07-17T11:07:26.1655652Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1655865Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1656210Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1656414Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1656913Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1657201Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1657413Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1657417Z 
2025-07-17T11:07:26.1657513Z SIGABRT(6), PID: 1800824, Thread 1805170: 
2025-07-17T11:07:26.1658063Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1658287Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1658492Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1658697Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1659155Z frame #4: <unknown function> + 0x5a02023 (0x7f3c585ad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1659736Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3c585b11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1660335Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3c585a81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1660730Z frame #7: <unknown function> + 0x79d655 (0x7f3c69a98655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1660967Z frame #8: <unknown function> + 0xdc253 (0x7f3c20ae3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1661183Z frame #9: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1661400Z frame #10: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1661410Z 
2025-07-17T11:07:26.1661504Z SIGABRT(6), PID: 1800823, Thread 1801182: 
2025-07-17T11:07:26.1661986Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1662194Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1662408Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1662612Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1663133Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1663348Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1663566Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1663570Z 
2025-07-17T11:07:26.1663663Z SIGABRT(6), PID: 1800822, Thread 1801132: 
2025-07-17T11:07:26.1664146Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1664355Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1664563Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1664847Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1665352Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1665564Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1665845Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1665850Z 
2025-07-17T11:07:26.1665946Z SIGABRT(6), PID: 1800818, Thread 1800856: 
2025-07-17T11:07:26.1666487Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1666700Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1666916Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1667115Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1667675Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1676384Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1676693Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1676699Z 
2025-07-17T11:07:26.1676810Z SIGABRT(6), PID: 1800823, Thread 1801187: 
2025-07-17T11:07:26.1677345Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1677578Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1677797Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1678007Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1678536Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1678748Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1678962Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1678968Z 
2025-07-17T11:07:26.1679067Z SIGABRT(6), PID: 1800822, Thread 1801135: 
2025-07-17T11:07:26.1679557Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1679780Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1679989Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1680196Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1680708Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1680918Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1681136Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1681140Z 
2025-07-17T11:07:26.1681246Z SIGABRT(6), PID: 1800820, Thread 1805156: 
2025-07-17T11:07:26.1681723Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1682085Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1682299Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1682506Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1682958Z frame #4: <unknown function> + 0x5a02023 (0x7f2b36130023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1683538Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2b361341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1684192Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2b3612b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1684591Z frame #7: <unknown function> + 0x79d655 (0x7f2b4761b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1684920Z frame #8: <unknown function> + 0xdc253 (0x7f2afe666253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1685139Z frame #9: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1685362Z frame #10: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1685366Z 
2025-07-17T11:07:26.1685465Z SIGABRT(6), PID: 1800818, Thread 1800857: 
2025-07-17T11:07:26.1685949Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1686163Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1686370Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1686573Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1687090Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1687318Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1687530Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1687534Z 
2025-07-17T11:07:26.1687627Z SIGABRT(6), PID: 1800823, Thread 1801191: 
2025-07-17T11:07:26.1688103Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1688314Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1688516Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1688721Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1689216Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1689427Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1689641Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1689646Z 
2025-07-17T11:07:26.1689743Z SIGABRT(6), PID: 1800822, Thread 1801140: 
2025-07-17T11:07:26.1690214Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1690498Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1690701Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1690907Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1691460Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1691674Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1691887Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1691947Z 
2025-07-17T11:07:26.1692040Z SIGABRT(6), PID: 1800818, Thread 1800858: 
2025-07-17T11:07:26.1692527Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1692735Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1693009Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1693206Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1693702Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1693907Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1694120Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1694123Z 
2025-07-17T11:07:26.1694214Z SIGABRT(6), PID: 1800823, Thread 1801195: 
2025-07-17T11:07:26.1694690Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1694899Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1695116Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1695315Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1695811Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1696103Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1696316Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1696322Z 
2025-07-17T11:07:26.1696411Z SIGABRT(6), PID: 1800822, Thread 1801144: 
2025-07-17T11:07:26.1696884Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1697090Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1697307Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1697508Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1698011Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1698220Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1698521Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1698526Z 
2025-07-17T11:07:26.1698615Z SIGABRT(6), PID: 1800818, Thread 1800860: 
2025-07-17T11:07:26.1699090Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1699297Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1699576Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1699781Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1700353Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1700564Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1700778Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1700782Z 
2025-07-17T11:07:26.1700871Z SIGABRT(6), PID: 1800823, Thread 1801199: 
2025-07-17T11:07:26.1701416Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1701631Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1701834Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1702035Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1702534Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1702745Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1702954Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1702960Z 
2025-07-17T11:07:26.1703057Z SIGABRT(6), PID: 1800822, Thread 1801147: 
2025-07-17T11:07:26.1703523Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1703731Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1703932Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1704133Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1704647Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1704861Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1705072Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1705078Z 
2025-07-17T11:07:26.1705170Z SIGABRT(6), PID: 1800818, Thread 1800863: 
2025-07-17T11:07:26.1705638Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1705845Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1706048Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1706247Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1706815Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1707028Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1707241Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1707245Z 
2025-07-17T11:07:26.1707340Z SIGABRT(6), PID: 1800823, Thread 1801203: 
2025-07-17T11:07:26.1707867Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1708081Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1708340Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1708544Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1709035Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1709313Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1709532Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1709536Z 
2025-07-17T11:07:26.1709639Z SIGABRT(6), PID: 1800822, Thread 1801151: 
2025-07-17T11:07:26.1710118Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1710331Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1710538Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1710739Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1711243Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1711455Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1711663Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1711671Z 
2025-07-17T11:07:26.1711760Z SIGABRT(6), PID: 1800818, Thread 1800865: 
2025-07-17T11:07:26.1712250Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1712463Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1712671Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1712871Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1713369Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1713575Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1713792Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1713796Z 
2025-07-17T11:07:26.1713885Z SIGABRT(6), PID: 1800823, Thread 1801207: 
2025-07-17T11:07:26.1714362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1714572Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1714856Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1715054Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1715552Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1715816Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1716028Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1716033Z 
2025-07-17T11:07:26.1716122Z SIGABRT(6), PID: 1800824, Thread 1805171: 
2025-07-17T11:07:26.1716673Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3c22876b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1716886Z frame #1: <unknown function> + 0x42520 (0x7f3d1412e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1717096Z frame #2: <unknown function> + 0x91117 (0x7f3d1417d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1717365Z frame #3: pthread_cond_wait + 0x211 (0x7f3d1417fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1717751Z frame #4: <unknown function> + 0x5a02023 (0x7f3c585ad023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1718322Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3c585b11ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1718918Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3c585a81d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1719300Z frame #7: <unknown function> + 0x79d655 (0x7f3c69a98655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1719537Z frame #8: <unknown function> + 0xdc253 (0x7f3c20ae3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1719748Z frame #9: <unknown function> + 0x94ac3 (0x7f3d14180ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1719970Z frame #10: <unknown function> + 0x126850 (0x7f3d14212850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1719974Z 
2025-07-17T11:07:26.1720068Z SIGABRT(6), PID: 1800822, Thread 1801154: 
2025-07-17T11:07:26.1720543Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1720752Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1720957Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1721156Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1721660Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1721868Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1722083Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1722087Z 
2025-07-17T11:07:26.1722192Z SIGABRT(6), PID: 1800818, Thread 1800867: 
2025-07-17T11:07:26.1722670Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1722878Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1723277Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1723483Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1723978Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1724191Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1724465Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1724470Z 
2025-07-17T11:07:26.1724565Z SIGABRT(6), PID: 1800823, Thread 1801212: 
2025-07-17T11:07:26.1725103Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1725313Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1725520Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1725723Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1726278Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1726491Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1726696Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1726701Z 
2025-07-17T11:07:26.1726794Z SIGABRT(6), PID: 1800822, Thread 1801157: 
2025-07-17T11:07:26.1727268Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1727484Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1727690Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1727892Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1728388Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1728597Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1728805Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1728809Z 
2025-07-17T11:07:26.1728902Z SIGABRT(6), PID: 1800820, Thread 1805157: 
2025-07-17T11:07:26.1729363Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1729574Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1729797Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1730003Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1730383Z frame #4: <unknown function> + 0x5a02023 (0x7f2b36130023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1730960Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2b361341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1731554Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2b3612b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1732012Z frame #7: <unknown function> + 0x79d655 (0x7f2b4761b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1732253Z frame #8: <unknown function> + 0xdc253 (0x7f2afe666253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1732464Z frame #9: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1732743Z frame #10: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1732748Z 
2025-07-17T11:07:26.1732839Z SIGABRT(6), PID: 1800823, Thread 1801216: 
2025-07-17T11:07:26.1733314Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1733577Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1733784Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1733981Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1734542Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1734769Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1734988Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1734993Z 
2025-07-17T11:07:26.1735086Z SIGABRT(6), PID: 1800818, Thread 1800869: 
2025-07-17T11:07:26.1735559Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1735766Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1736056Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1736256Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1736755Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1736962Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1737173Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1737177Z 
2025-07-17T11:07:26.1737266Z SIGABRT(6), PID: 1800822, Thread 1801160: 
2025-07-17T11:07:26.1737743Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1737947Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1738152Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1738352Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1738850Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1739054Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1739266Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1739271Z 
2025-07-17T11:07:26.1739360Z SIGABRT(6), PID: 1800823, Thread 1801220: 
2025-07-17T11:07:26.1739853Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1740149Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1740360Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1740560Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1741144Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1741356Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1741637Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1741641Z 
2025-07-17T11:07:26.1741736Z SIGABRT(6), PID: 1800818, Thread 1800870: 
2025-07-17T11:07:26.1742212Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1742434Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1742760Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1742961Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1743457Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1743675Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1743885Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1743893Z 
2025-07-17T11:07:26.1743986Z SIGABRT(6), PID: 1800822, Thread 1801163: 
2025-07-17T11:07:26.1744452Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1744659Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1744862Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1745069Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1745566Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1745778Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1745987Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1745993Z 
2025-07-17T11:07:26.1746086Z SIGABRT(6), PID: 1800823, Thread 1801224: 
2025-07-17T11:07:26.1746552Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1746761Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1746963Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1747163Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1747669Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1747880Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1748160Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1748164Z 
2025-07-17T11:07:26.1748262Z SIGABRT(6), PID: 1800818, Thread 1800872: 
2025-07-17T11:07:26.1748740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1748952Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1749237Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1749441Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1749996Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1750221Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1750431Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1750436Z 
2025-07-17T11:07:26.1750592Z SIGABRT(6), PID: 1800822, Thread 1801167: 
2025-07-17T11:07:26.1751065Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1751274Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1751478Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1751679Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1752176Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1752405Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1752616Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1752624Z 
2025-07-17T11:07:26.1752716Z SIGABRT(6), PID: 1800823, Thread 1801228: 
2025-07-17T11:07:26.1753195Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1753405Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1753615Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1753814Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1754313Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1754522Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1754731Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1754735Z 
2025-07-17T11:07:26.1754824Z SIGABRT(6), PID: 1800818, Thread 1800874: 
2025-07-17T11:07:26.1755303Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1755509Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1755713Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1755912Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1756482Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1756694Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1756905Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1756909Z 
2025-07-17T11:07:26.1756998Z SIGABRT(6), PID: 1800823, Thread 1801232: 
2025-07-17T11:07:26.1757534Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1757753Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1758013Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1758213Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1758714Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1758989Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1759200Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1759291Z SIGABRT(6), PID: 1800822, Thread 1801171: 
2025-07-17T11:07:26.1759765Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1759974Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1760187Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1760391Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1760889Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1761098Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1761309Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1761313Z 
2025-07-17T11:07:26.1761316Z 
2025-07-17T11:07:26.1761405Z SIGABRT(6), PID: 1800818, Thread 1800876: 
2025-07-17T11:07:26.1761876Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1762082Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1762293Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1762493Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1762997Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1763205Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1763427Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1763431Z 
2025-07-17T11:07:26.1763522Z SIGABRT(6), PID: 1800822, Thread 1801175: 
2025-07-17T11:07:26.1763992Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1764199Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1764472Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1764672Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1765179Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1765452Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1765666Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1765758Z SIGABRT(6), PID: 1800823, Thread 1801236: 
2025-07-17T11:07:26.1766286Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1766499Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1766699Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1766956Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1767449Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1767663Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1767874Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1767878Z 
2025-07-17T11:07:26.1767882Z 
2025-07-17T11:07:26.1767989Z SIGABRT(6), PID: 1800818, Thread 1800878: 
2025-07-17T11:07:26.1768464Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1768680Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1768888Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1769096Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1769590Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1769816Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1770037Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1770042Z 
2025-07-17T11:07:26.1770146Z SIGABRT(6), PID: 1800822, Thread 1801179: 
2025-07-17T11:07:26.1770616Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1770825Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1771037Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1771241Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1771739Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1771957Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1772165Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1772170Z 
2025-07-17T11:07:26.1772338Z SIGABRT(6), PID: 1800823, Thread 1801240: 
2025-07-17T11:07:26.1772811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1773025Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1773227Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1773493Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1773996Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1774271Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1774490Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1774496Z 
2025-07-17T11:07:26.1774588Z SIGABRT(6), PID: 1800820, Thread 1805158: 
2025-07-17T11:07:26.1775083Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1775359Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1775578Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1775781Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1776220Z frame #4: <unknown function> + 0x5a02023 (0x7f2b36130023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1776791Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2b361341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1777397Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2b3612b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1777785Z frame #7: <unknown function> + 0x79d655 (0x7f2b4761b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1778034Z frame #8: <unknown function> + 0xdc253 (0x7f2afe666253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1778245Z frame #9: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1778468Z frame #10: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1778473Z 
2025-07-17T11:07:26.1778565Z SIGABRT(6), PID: 1800822, Thread 1801183: 
2025-07-17T11:07:26.1779042Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1779251Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1779465Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1779667Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1780193Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1780409Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1780628Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1780632Z 
2025-07-17T11:07:26.1780722Z SIGABRT(6), PID: 1800818, Thread 1800880: 
2025-07-17T11:07:26.1781306Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1781513Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1781726Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1782004Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1782507Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1782718Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1783004Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1783101Z SIGABRT(6), PID: 1800823, Thread 1801243: 
2025-07-17T11:07:26.1783589Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1783875Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1784086Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1784285Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1784788Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1785002Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1785225Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1785231Z 
2025-07-17T11:07:26.1785235Z 
2025-07-17T11:07:26.1785336Z SIGABRT(6), PID: 1800822, Thread 1801186: 
2025-07-17T11:07:26.1785805Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1786017Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1786223Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1786430Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1786925Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1787140Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1787352Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1787356Z 
2025-07-17T11:07:26.1787462Z SIGABRT(6), PID: 1800818, Thread 1800882: 
2025-07-17T11:07:26.1787938Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1788153Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1788358Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1788567Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1789063Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1789351Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1789561Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1789565Z 
2025-07-17T11:07:26.1789666Z SIGABRT(6), PID: 1800823, Thread 1801246: 
2025-07-17T11:07:26.1790147Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1790422Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1790631Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1790839Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1791390Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1791608Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1791820Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1791883Z 
2025-07-17T11:07:26.1791983Z SIGABRT(6), PID: 1800822, Thread 1801190: 
2025-07-17T11:07:26.1792457Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1792681Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1792893Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1793104Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1793600Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1793814Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1794024Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1794043Z 
2025-07-17T11:07:26.1794133Z SIGABRT(6), PID: 1800818, Thread 1800884: 
2025-07-17T11:07:26.1794606Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1794813Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1795023Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1795225Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1795734Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1795943Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1796159Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1796163Z 
2025-07-17T11:07:26.1796253Z SIGABRT(6), PID: 1800823, Thread 1801249: 
2025-07-17T11:07:26.1796727Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1796933Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1797146Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1797344Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1797944Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1798158Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1798368Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1798372Z 
2025-07-17T11:07:26.1798520Z SIGABRT(6), PID: 1800822, Thread 1801194: 
2025-07-17T11:07:26.1799000Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1799266Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1799480Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1799684Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1800187Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1800475Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1800698Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1800702Z 
2025-07-17T11:07:26.1800795Z SIGABRT(6), PID: 1800818, Thread 1800886: 
2025-07-17T11:07:26.1801268Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1801480Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1801698Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1801900Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1802405Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1802617Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1802842Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1802846Z 
2025-07-17T11:07:26.1802941Z SIGABRT(6), PID: 1800823, Thread 1801251: 
2025-07-17T11:07:26.1803426Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1803632Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1803845Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1804042Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1804542Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1804753Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1804968Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1804971Z 
2025-07-17T11:07:26.1805070Z SIGABRT(6), PID: 1800822, Thread 1801198: 
2025-07-17T11:07:26.1805542Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1805830Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1806032Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1806241Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1806807Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1807030Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1807246Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1807252Z 
2025-07-17T11:07:26.1807416Z SIGABRT(6), PID: 1800823, Thread 1801254: 
2025-07-17T11:07:26.1807891Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1808109Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1808311Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1808598Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1809097Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1809309Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1809518Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1809524Z 
2025-07-17T11:07:26.1809619Z SIGABRT(6), PID: 1800818, Thread 1800889: 
2025-07-17T11:07:26.1810098Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1810320Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1810524Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1810728Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1811225Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1811438Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1811645Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1811649Z 
2025-07-17T11:07:26.1811746Z SIGABRT(6), PID: 1800822, Thread 1801202: 
2025-07-17T11:07:26.1812213Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1812421Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1812629Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1812842Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1813338Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1813554Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1813763Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1813834Z 
2025-07-17T11:07:26.1813933Z SIGABRT(6), PID: 1800823, Thread 1801257: 
2025-07-17T11:07:26.1814404Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1814617Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1814884Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1815104Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1815601Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1815872Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1816171Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1816175Z 
2025-07-17T11:07:26.1816267Z SIGABRT(6), PID: 1800818, Thread 1800891: 
2025-07-17T11:07:26.1816747Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1817034Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1817245Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1817449Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1817957Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1818168Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1818387Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1818391Z 
2025-07-17T11:07:26.1818482Z SIGABRT(6), PID: 1800820, Thread 1805159: 
2025-07-17T11:07:26.1818957Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1819167Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1819374Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1819571Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1819962Z frame #4: <unknown function> + 0x5a02023 (0x7f2b36130023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1820560Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2b361341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1821156Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2b3612b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1821541Z frame #7: <unknown function> + 0x79d655 (0x7f2b4761b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1821781Z frame #8: <unknown function> + 0xdc253 (0x7f2afe666253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1821989Z frame #9: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1822211Z frame #10: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1822295Z 
2025-07-17T11:07:26.1822394Z SIGABRT(6), PID: 1800822, Thread 1801206: 
2025-07-17T11:07:26.1822892Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1823119Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1823336Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1823610Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1824147Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1824436Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1824662Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1824669Z 
2025-07-17T11:07:26.1824762Z SIGABRT(6), PID: 1800823, Thread 1801260: 
2025-07-17T11:07:26.1825258Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1825554Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1825766Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1825974Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1826481Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1826696Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1826906Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1826910Z 
2025-07-17T11:07:26.1827007Z SIGABRT(6), PID: 1800818, Thread 1800893: 
2025-07-17T11:07:26.1827481Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1827694Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1827900Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1828106Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1828606Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1828821Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1829030Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1829034Z 
2025-07-17T11:07:26.1829134Z SIGABRT(6), PID: 1800822, Thread 1801210: 
2025-07-17T11:07:26.1829601Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1829816Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1830029Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1830231Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1830742Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1831035Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1831243Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1831343Z SIGABRT(6), PID: 1800823, Thread 1801263: 
2025-07-17T11:07:26.1831872Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1832096Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1832301Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1832506Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1833056Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1833285Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1833499Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1833562Z 
2025-07-17T11:07:26.1833565Z 
2025-07-17T11:07:26.1833669Z SIGABRT(6), PID: 1800818, Thread 1800895: 
2025-07-17T11:07:26.1834141Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1834351Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1834555Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1834761Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1835254Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1835475Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1835693Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1835712Z 
2025-07-17T11:07:26.1835806Z SIGABRT(6), PID: 1800823, Thread 1801266: 
2025-07-17T11:07:26.1836279Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1836485Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1836698Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1836895Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1837400Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1837609Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1837825Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1837829Z 
2025-07-17T11:07:26.1837929Z SIGABRT(6), PID: 1800822, Thread 1801213: 
2025-07-17T11:07:26.1838411Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1838618Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1838826Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1839098Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1839601Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1839811Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1840081Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1840086Z 
2025-07-17T11:07:26.1840176Z SIGABRT(6), PID: 1800823, Thread 1801269: 
2025-07-17T11:07:26.1840653Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1840941Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1841161Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1841360Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1841859Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1842130Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1842345Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1842437Z SIGABRT(6), PID: 1800818, Thread 1800897: 
2025-07-17T11:07:26.1842967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1843216Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1843433Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1843634Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1844135Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1844345Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1844562Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1844566Z 
2025-07-17T11:07:26.1844570Z 
2025-07-17T11:07:26.1844660Z SIGABRT(6), PID: 1800822, Thread 1801217: 
2025-07-17T11:07:26.1845138Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1845346Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1845564Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1845766Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1846268Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1846475Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1846691Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1846694Z 
2025-07-17T11:07:26.1846788Z SIGABRT(6), PID: 1800823, Thread 1801272: 
2025-07-17T11:07:26.1847264Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1847550Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1847755Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1847963Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1848525Z frame #4: <unknown function> + 0x3475ab (0x7f3a904745ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1848737Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1848944Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1849006Z 
2025-07-17T11:07:26.1849112Z SIGABRT(6), PID: 1800822, Thread 1801221: 
2025-07-17T11:07:26.1849583Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1849797Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1850178Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1850399Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1850901Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1851115Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1851330Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1851427Z SIGABRT(6), PID: 1800818, Thread 1800899: 
2025-07-17T11:07:26.1851894Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1852106Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1852312Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1852518Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1853007Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1853234Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1853448Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1853454Z 
2025-07-17T11:07:26.1853457Z 
2025-07-17T11:07:26.1853550Z SIGABRT(6), PID: 1800823, Thread 1801350: 
2025-07-17T11:07:26.1854017Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1854484Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f39a1cd308a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1854696Z frame #2: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1854896Z frame #3: pthread_kill + 0x12c (0x7f3a935de9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1855065Z frame #4: raise + 0x16 (0x7f3a9358a476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1855234Z frame #5: abort + 0xd3 (0x7f3a935707f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1855482Z frame #6: <unknown function> + 0x972cc (0x7f390f85c2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1855800Z frame #7: <unknown function> + 0x95f47 (0x7f390f85af47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1856106Z frame #8: <unknown function> + 0x34b71 (0x7f390f7f9b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1856326Z frame #9: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1856542Z frame #10: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1856624Z 
2025-07-17T11:07:26.1856728Z SIGABRT(6), PID: 1800822, Thread 1801225: 
2025-07-17T11:07:26.1857208Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1857498Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1857706Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1857919Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1858425Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1858711Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1858925Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1858938Z 
2025-07-17T11:07:26.1859037Z SIGABRT(6), PID: 1800818, Thread 1800900: 
2025-07-17T11:07:26.1859519Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1859735Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1859953Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1860155Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1860679Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1860890Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1861103Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1861107Z 
2025-07-17T11:07:26.1861201Z SIGABRT(6), PID: 1800820, Thread 1805160: 
2025-07-17T11:07:26.1861681Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1861892Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1862102Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1862301Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1862691Z frame #4: <unknown function> + 0x5a02023 (0x7f2b36130023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1863262Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2b361341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1863864Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2b3612b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1864333Z frame #7: <unknown function> + 0x79d655 (0x7f2b4761b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1864577Z frame #8: <unknown function> + 0xdc253 (0x7f2afe666253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1864790Z frame #9: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1865016Z frame #10: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1865019Z 
2025-07-17T11:07:26.1865170Z SIGABRT(6), PID: 1800822, Thread 1801229: 
2025-07-17T11:07:26.1865654Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1865861Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1866133Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1866341Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1866852Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1867127Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1867367Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1867459Z SIGABRT(6), PID: 1800823, Thread 1804874: 
2025-07-17T11:07:26.1867942Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1868153Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1868330Z frame #2: __poll + 0x4f (0x7f3a93660bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1868741Z frame #3: <unknown function> + 0x489f (0x7f3a9316c89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.1868979Z frame #4: <unknown function> + 0x1483bc (0x556da326c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1869231Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1869464Z frame #6: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1869703Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1869933Z frame #8: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1870180Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x556da3254a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1870417Z frame #10: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1870665Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x556da3252c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1870900Z frame #12: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1871144Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1871376Z frame #14: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1871612Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1871842Z frame #16: <unknown function> + 0x147734 (0x556da326b734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1872076Z frame #17: <unknown function> + 0x21ca82 (0x556da3340a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1872305Z frame #18: <unknown function> + 0x21c9e4 (0x556da33409e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1872588Z frame #19: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1872811Z frame #20: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1872817Z 
2025-07-17T11:07:26.1872820Z 
2025-07-17T11:07:26.1872913Z SIGABRT(6), PID: 1800818, Thread 1800902: 
2025-07-17T11:07:26.1873482Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1873703Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1873914Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1874183Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1874694Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1874914Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1875183Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1875187Z 
2025-07-17T11:07:26.1875285Z SIGABRT(6), PID: 1800822, Thread 1801233: 
2025-07-17T11:07:26.1875758Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1875973Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1876190Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1876400Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1876905Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1877120Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1877327Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1877331Z 
2025-07-17T11:07:26.1877432Z SIGABRT(6), PID: 1800818, Thread 1800903: 
2025-07-17T11:07:26.1877904Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1878121Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1878333Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1878541Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1879040Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1879260Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1879475Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1879479Z 
2025-07-17T11:07:26.1879583Z SIGABRT(6), PID: 1800822, Thread 1801237: 
2025-07-17T11:07:26.1880053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1880279Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1880558Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1880763Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1881260Z frame #4: <unknown function> + 0x3475ab (0x7f22d1e535ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1881482Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1881754Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1881761Z 
2025-07-17T11:07:26.1881863Z SIGABRT(6), PID: 1800818, Thread 1800905: 
2025-07-17T11:07:26.1882414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1882644Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1882852Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1883062Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1883621Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1883838Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1884054Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1884064Z 
2025-07-17T11:07:26.1884157Z SIGABRT(6), PID: 1800823, Thread 1804875: 
2025-07-17T11:07:26.1884645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1884853Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1885067Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1885293Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f3a935dc2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1885750Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f39cda1b048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1886177Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f39cda1dbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1886427Z frame #6: <unknown function> + 0xdc253 (0x7f399ff3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1886638Z frame #7: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1886858Z frame #8: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1886954Z SIGABRT(6), PID: 1800822, Thread 1801346: 
2025-07-17T11:07:26.1887438Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1887891Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f21e36af08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1888109Z frame #2: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1888398Z frame #3: pthread_kill + 0x12c (0x7f22d4fbd9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1888578Z frame #4: raise + 0x16 (0x7f22d4f69476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1888741Z frame #5: abort + 0xd3 (0x7f22d4f4f7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1888981Z frame #6: <unknown function> + 0x972cc (0x7f21512382cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1889285Z frame #7: <unknown function> + 0x95f47 (0x7f2151236f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1889520Z frame #8: <unknown function> + 0x34b71 (0x7f21511d5b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1889736Z frame #9: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1890013Z frame #10: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1890018Z 
2025-07-17T11:07:26.1890021Z 
2025-07-17T11:07:26.1890117Z SIGABRT(6), PID: 1800818, Thread 1800907: 
2025-07-17T11:07:26.1890603Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1890873Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1891094Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1891299Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1891807Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1892080Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1892302Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1892305Z 
2025-07-17T11:07:26.1892401Z SIGABRT(6), PID: 1800822, Thread 1804470: 
2025-07-17T11:07:26.1892885Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1893093Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1893270Z frame #2: __poll + 0x4f (0x7f22d503fbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1893683Z frame #3: <unknown function> + 0x489f (0x7f22d4b4b89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.1893918Z frame #4: <unknown function> + 0x1483bc (0x562c7761a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1894168Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1894401Z frame #6: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1894651Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1894888Z frame #8: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1895133Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x562c77602a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1895376Z frame #10: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1895623Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x562c77600c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1895861Z frame #12: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1896172Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1896407Z frame #14: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1896650Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1896883Z frame #16: <unknown function> + 0x147734 (0x562c77619734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1897106Z frame #17: <unknown function> + 0x21ca82 (0x562c776eea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1897419Z frame #18: <unknown function> + 0x21c9e4 (0x562c776ee9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1897635Z frame #19: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1897865Z frame #20: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1897870Z 
2025-07-17T11:07:26.1897966Z SIGABRT(6), PID: 1800823, Thread 1804876: 
2025-07-17T11:07:26.1898519Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1898742Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1899032Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1899264Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f3a935dc2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1899756Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f39cda15741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1900060Z frame #5: <unknown function> + 0xdc253 (0x7f399ff3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1900275Z frame #6: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1900488Z frame #7: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1900499Z 
2025-07-17T11:07:26.1900592Z SIGABRT(6), PID: 1800818, Thread 1800909: 
2025-07-17T11:07:26.1901080Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1901288Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1901503Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1901710Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1902218Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1902434Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1902650Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1902654Z 
2025-07-17T11:07:26.1902749Z SIGABRT(6), PID: 1800823, Thread 1804877: 
2025-07-17T11:07:26.1903234Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1903443Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1903619Z frame #2: ioctl + 0x3f (0x7f3a9366294f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1903858Z frame #3: <unknown function> + 0x13eef0 (0x7f390f903ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1904101Z frame #4: <unknown function> + 0x1378c4 (0x7f390f8fc8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1904337Z frame #5: <unknown function> + 0x95b2b (0x7f390f85ab2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1904572Z frame #6: <unknown function> + 0x34b71 (0x7f390f7f9b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1904782Z frame #7: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1905003Z frame #8: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1905006Z 
2025-07-17T11:07:26.1905099Z SIGABRT(6), PID: 1800820, Thread 1805161: 
2025-07-17T11:07:26.1905674Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1905883Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1906097Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1906360Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1906755Z frame #4: <unknown function> + 0x5a02023 (0x7f2b36130023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1907403Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2b361341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1908011Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2b3612b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1908400Z frame #7: <unknown function> + 0x79d655 (0x7f2b4761b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1908702Z frame #8: <unknown function> + 0xdc253 (0x7f2afe666253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1908920Z frame #9: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1909148Z frame #10: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1909152Z 
2025-07-17T11:07:26.1909249Z SIGABRT(6), PID: 1800818, Thread 1800911: 
2025-07-17T11:07:26.1909731Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1909942Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1910164Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1910366Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1910878Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1911089Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1911313Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1911317Z 
2025-07-17T11:07:26.1911423Z SIGABRT(6), PID: 1800823, Thread 1805098: 
2025-07-17T11:07:26.1911905Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1912121Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1912287Z frame #2: __poll + 0x4f (0x7f3a93660bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1912499Z frame #3: <unknown function> + 0xd4540 (0x7f3957a73540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1912714Z frame #4: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1912926Z frame #5: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1912937Z 
2025-07-17T11:07:26.1913030Z SIGABRT(6), PID: 1800822, Thread 1804471: 
2025-07-17T11:07:26.1913507Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1913718Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1914003Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1914227Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f22d4fbb2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1914686Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f220f3f7048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1915172Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f220f3f9bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1915413Z frame #6: <unknown function> + 0xdc253 (0x7f21e191b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1915680Z frame #7: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1915915Z frame #8: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1915922Z 
2025-07-17T11:07:26.1916014Z SIGABRT(6), PID: 1800818, Thread 1800915: 
2025-07-17T11:07:26.1916489Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1916760Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1916976Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1917178Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1917685Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1917903Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1918131Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1918135Z 
2025-07-17T11:07:26.1918228Z SIGABRT(6), PID: 1800823, Thread 1805106: 
2025-07-17T11:07:26.1918707Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1918915Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1919088Z frame #2: __poll + 0x4f (0x7f3a93660bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1919302Z frame #3: <unknown function> + 0xd65a2 (0x7f3957a755a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1919519Z frame #4: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1919733Z frame #5: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1919739Z 
2025-07-17T11:07:26.1919837Z SIGABRT(6), PID: 1800823, Thread 1805114: 
2025-07-17T11:07:26.1920334Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1920555Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1920763Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1920975Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1921192Z frame #4: <unknown function> + 0xd1e7f (0x7f3957a70e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1921411Z frame #5: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1921622Z frame #6: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1921626Z 
2025-07-17T11:07:26.1921795Z SIGABRT(6), PID: 1800822, Thread 1804472: 
2025-07-17T11:07:26.1922268Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1922487Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1922696Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1922989Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f22d4fbb2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1923469Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f220f3f1741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1923768Z frame #5: <unknown function> + 0xdc253 (0x7f21e191b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1923987Z frame #6: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1924208Z frame #7: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1924212Z 
2025-07-17T11:07:26.1924303Z SIGABRT(6), PID: 1800818, Thread 1800919: 
2025-07-17T11:07:26.1924839Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1925059Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1925263Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1925475Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1925981Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1926202Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1926413Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1926417Z 
2025-07-17T11:07:26.1926527Z SIGABRT(6), PID: 1800822, Thread 1804475: 
2025-07-17T11:07:26.1926995Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1927216Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1927384Z frame #2: ioctl + 0x3f (0x7f22d504194f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1927630Z frame #3: <unknown function> + 0x13eef0 (0x7f21512dfef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1927864Z frame #4: <unknown function> + 0x1378c4 (0x7f21512d88c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1928109Z frame #5: <unknown function> + 0x95b2b (0x7f2151236b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1928336Z frame #6: <unknown function> + 0x34b71 (0x7f21511d5b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1928557Z frame #7: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1928776Z frame #8: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1928876Z SIGABRT(6), PID: 1800818, Thread 1800924: 
2025-07-17T11:07:26.1929347Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1929560Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1929766Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1929975Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1930546Z frame #4: <unknown function> + 0x3475ab (0x7fa2748675ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.1930778Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1930986Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1931148Z SIGABRT(6), PID: 1800823, Thread 1805116: 
2025-07-17T11:07:26.1931624Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1931846Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1932116Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1932336Z frame #3: <unknown function> + 0x9ca53 (0x7f3a935e4a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1932595Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x556da3238bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1932917Z frame #5: <unknown function> + 0x1d314f (0x556da32f714f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1933142Z frame #6: <unknown function> + 0x1d2f6b (0x556da32f6f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1933370Z frame #7: <unknown function> + 0x6df36 (0x556da3191f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1933615Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1933850Z frame #9: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1934092Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1934328Z frame #11: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1934567Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1934799Z frame #13: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1935040Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1935273Z frame #15: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1935512Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1935741Z frame #17: <unknown function> + 0x147734 (0x556da326b734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1936046Z frame #18: <unknown function> + 0x21ca82 (0x556da3340a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1936276Z frame #19: <unknown function> + 0x21c9e4 (0x556da33409e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1936493Z frame #20: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1936715Z frame #21: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1936721Z 
2025-07-17T11:07:26.1936726Z 
2025-07-17T11:07:26.1936729Z 
2025-07-17T11:07:26.1936820Z SIGABRT(6), PID: 1800820, Thread 1805162: 
2025-07-17T11:07:26.1937345Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1937560Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1937776Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1937982Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1938455Z frame #4: <unknown function> + 0x5a02023 (0x7f2b36130023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1939029Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2b361341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1939713Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2b3612b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1940101Z frame #7: <unknown function> + 0x79d655 (0x7f2b4761b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1940411Z frame #8: <unknown function> + 0xdc253 (0x7f2afe666253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1940628Z frame #9: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1940853Z frame #10: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1940857Z 
2025-07-17T11:07:26.1940948Z SIGABRT(6), PID: 1800818, Thread 1801343: 
2025-07-17T11:07:26.1941526Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1941992Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fa1860c608a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1942204Z frame #2: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1942405Z frame #3: pthread_kill + 0x12c (0x7fa2779d19fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1942572Z frame #4: raise + 0x16 (0x7fa27797d476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1942741Z frame #5: abort + 0xd3 (0x7fa2779637f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1942978Z frame #6: <unknown function> + 0x972cc (0x7fa0f3c4f2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1943214Z frame #7: <unknown function> + 0x95f47 (0x7fa0f3c4df47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1943445Z frame #8: <unknown function> + 0x34b71 (0x7fa0f3becb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1943671Z frame #9: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1943889Z frame #10: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1943893Z 
2025-07-17T11:07:26.1943997Z SIGABRT(6), PID: 1800822, Thread 1805095: 
2025-07-17T11:07:26.1944479Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1944714Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1944888Z frame #2: __poll + 0x4f (0x7f22d503fbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1945118Z frame #3: <unknown function> + 0xd4540 (0x7f219944f540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1945331Z frame #4: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1945548Z frame #5: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1945554Z 
2025-07-17T11:07:26.1945648Z SIGABRT(6), PID: 1800823, Thread 1805127: 
2025-07-17T11:07:26.1946138Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1946348Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1946522Z frame #2: read + 0x4c (0x7f3a9365c81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1946784Z frame #3: _Py_read + 0x4c (0x556da32fea6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1947020Z frame #4: <unknown function> + 0x208c3b (0x556da332cc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1947245Z frame #5: <unknown function> + 0x14648d (0x556da326a48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1947501Z frame #6: PyObject_VectorcallMethod + 0x85 (0x556da3279215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1947792Z frame #7: <unknown function> + 0x208ae7 (0x556da332cae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1948041Z frame #8: <unknown function> + 0x208472 (0x556da332c472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1948266Z frame #9: <unknown function> + 0x1da6ee (0x556da32fe6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1948567Z frame #10: <unknown function> + 0x1483bc (0x556da326c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1948825Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1949060Z frame #12: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1949363Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x556da32503bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1949592Z frame #14: <unknown function> + 0x147734 (0x556da326b734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1949848Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x556da3252c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1950086Z frame #16: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1950323Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1950558Z frame #18: _PyFunction_Vectorcall + 0x6c (0x556da32600ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1950795Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x556da32507b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1951020Z frame #20: <unknown function> + 0x147734 (0x556da326b734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1951240Z frame #21: <unknown function> + 0x21ca82 (0x556da3340a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1951465Z frame #22: <unknown function> + 0x21c9e4 (0x556da33409e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1951679Z frame #23: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1951894Z frame #24: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1951898Z 
2025-07-17T11:07:26.1951994Z SIGABRT(6), PID: 1800818, Thread 1803664: 
2025-07-17T11:07:26.1952493Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1952706Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1952881Z frame #2: __poll + 0x4f (0x7fa277a53bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1953293Z frame #3: <unknown function> + 0x489f (0x7fa27755f89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.1953529Z frame #4: <unknown function> + 0x1483bc (0x5644aecc93bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1953774Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1954006Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1954243Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1954492Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1954812Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5644aecb1a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1955052Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1955304Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5644aecafc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1955536Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1955834Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1956068Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1956307Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1956597Z frame #16: <unknown function> + 0x147734 (0x5644aecc8734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1956835Z frame #17: <unknown function> + 0x21ca82 (0x5644aed9da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1957068Z frame #18: <unknown function> + 0x21c9e4 (0x5644aed9d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1957349Z frame #19: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1957573Z frame #20: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1957577Z 
2025-07-17T11:07:26.1957678Z SIGABRT(6), PID: 1800822, Thread 1805107: 
2025-07-17T11:07:26.1958160Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1958376Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1958554Z frame #2: __poll + 0x4f (0x7f22d503fbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1958775Z frame #3: <unknown function> + 0xd65a2 (0x7f21994515a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1959010Z frame #4: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1959224Z frame #5: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1959227Z 
2025-07-17T11:07:26.1959328Z SIGABRT(6), PID: 1800822, Thread 1805112: 
2025-07-17T11:07:26.1959797Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1960009Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1960217Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1960436Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1960657Z frame #4: <unknown function> + 0xd1e7f (0x7f219944ce7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.1960872Z frame #5: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1961087Z frame #6: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1961091Z 
2025-07-17T11:07:26.1961190Z SIGABRT(6), PID: 1800822, Thread 1805119: 
2025-07-17T11:07:26.1961660Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1961876Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1962083Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1962298Z frame #3: <unknown function> + 0x9ca53 (0x7f22d4fc3a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1962623Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x562c775e6bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1962857Z frame #5: <unknown function> + 0x1d314f (0x562c776a514f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1963083Z frame #6: <unknown function> + 0x1d2f6b (0x562c776a4f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1963323Z frame #7: <unknown function> + 0x6df36 (0x562c7753ff36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1963647Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1963888Z frame #9: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1964127Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1964445Z frame #11: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1964693Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1964928Z frame #13: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1965178Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1965480Z frame #15: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1965719Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1965951Z frame #17: <unknown function> + 0x147734 (0x562c77619734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1966176Z frame #18: <unknown function> + 0x21ca82 (0x562c776eea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1966408Z frame #19: <unknown function> + 0x21c9e4 (0x562c776ee9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1966622Z frame #20: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1966847Z frame #21: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1966851Z 
2025-07-17T11:07:26.1966948Z SIGABRT(6), PID: 1800822, Thread 1805136: 
2025-07-17T11:07:26.1967439Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1967653Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1967830Z frame #2: read + 0x4c (0x7f22d503b81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1968012Z frame #3: _Py_read + 0x4c (0x562c776aca6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1968255Z frame #4: <unknown function> + 0x208c3b (0x562c776dac3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1968475Z frame #5: <unknown function> + 0x14648d (0x562c7761848d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1968724Z frame #6: PyObject_VectorcallMethod + 0x85 (0x562c77627215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1968947Z frame #7: <unknown function> + 0x208ae7 (0x562c776daae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1969170Z frame #8: <unknown function> + 0x208472 (0x562c776da472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1969396Z frame #9: <unknown function> + 0x1da6ee (0x562c776ac6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1969629Z frame #10: <unknown function> + 0x1483bc (0x562c7761a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1969869Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1970108Z frame #12: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1970347Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x562c775fe3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1970647Z frame #14: <unknown function> + 0x147734 (0x562c77619734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1970916Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x562c77600c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1971177Z frame #16: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1971436Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1971730Z frame #18: _PyFunction_Vectorcall + 0x6c (0x562c7760e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1971972Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x562c775fe7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1972203Z frame #20: <unknown function> + 0x147734 (0x562c77619734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1972484Z frame #21: <unknown function> + 0x21ca82 (0x562c776eea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1972720Z frame #22: <unknown function> + 0x21c9e4 (0x562c776ee9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.1972939Z frame #23: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1973222Z frame #24: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1973227Z 
2025-07-17T11:07:26.1973320Z SIGABRT(6), PID: 1800820, Thread 1805163: 
2025-07-17T11:07:26.1973801Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f2b003f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1974024Z frame #1: <unknown function> + 0x42520 (0x7f2bf1cb1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1974233Z frame #2: <unknown function> + 0x91117 (0x7f2bf1d00117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1974452Z frame #3: pthread_cond_wait + 0x211 (0x7f2bf1d02a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1974834Z frame #4: <unknown function> + 0x5a02023 (0x7f2b36130023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1975442Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2b361341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1976138Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2b3612b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1976529Z frame #7: <unknown function> + 0x79d655 (0x7f2b4761b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1976764Z frame #8: <unknown function> + 0xdc253 (0x7f2afe666253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1976981Z frame #9: <unknown function> + 0x94ac3 (0x7f2bf1d03ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1977196Z frame #10: <unknown function> + 0x126850 (0x7f2bf1d95850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1977200Z 
2025-07-17T11:07:26.1977298Z SIGABRT(6), PID: 1800823, Thread 1805148: 
2025-07-17T11:07:26.1977778Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1977995Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1978202Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1978410Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1978786Z frame #4: <unknown function> + 0x5a02023 (0x7f39d7a09023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1979440Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f39d7a0d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1980037Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f39d7a041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1980495Z frame #7: <unknown function> + 0x79d655 (0x7f39e8ef4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1980734Z frame #8: <unknown function> + 0xdc253 (0x7f399ff3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1980959Z frame #9: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1981340Z frame #10: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1981346Z 
2025-07-17T11:07:26.1981445Z SIGABRT(6), PID: 1800818, Thread 1803665: 
2025-07-17T11:07:26.1981959Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1982249Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1982435Z frame #2: epoll_wait + 0x5e (0x7fa277a60e2e in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1982776Z frame #3: uv__io_poll + 0x6bd (0x7fa1c861c44d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1983095Z frame #4: uv_run + 0x12a (0x7fa1c861036a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1983482Z frame #5: <unknown function> + 0x658a08a (0x7fa1bc98408a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1983716Z frame #6: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1983944Z frame #7: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1984155Z frame #8: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1984166Z 
2025-07-17T11:07:26.1984260Z SIGABRT(6), PID: 1800818, Thread 1805079: 
2025-07-17T11:07:26.1984748Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1984957Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1985170Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1985395Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa2779cf2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1985850Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fa1b1e0e048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1986283Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fa1b1e10bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1986523Z frame #6: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1986731Z frame #7: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1986948Z frame #8: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1986953Z 
2025-07-17T11:07:26.1987048Z SIGABRT(6), PID: 1800822, Thread 1805174: 
2025-07-17T11:07:26.1987529Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1987807Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1988078Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1988285Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1988666Z frame #4: <unknown function> + 0x5a02023 (0x7f22193e5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1989296Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f22193e91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1989951Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f22193e01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1990343Z frame #7: <unknown function> + 0x79d655 (0x7f222a8d0655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1990580Z frame #8: <unknown function> + 0xdc253 (0x7f21e191b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1990851Z frame #9: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1991070Z frame #10: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1991074Z 
2025-07-17T11:07:26.1991168Z SIGABRT(6), PID: 1800818, Thread 1805080: 
2025-07-17T11:07:26.1991654Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1991862Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1992075Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1992320Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa2779cf2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1992802Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fa1b1e08741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.1993032Z frame #5: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1993251Z frame #6: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1993462Z frame #7: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1993467Z 
2025-07-17T11:07:26.1993566Z SIGABRT(6), PID: 1800818, Thread 1805082: 
2025-07-17T11:07:26.1994036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1994253Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1994425Z frame #2: ioctl + 0x3f (0x7fa277a5594f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1994675Z frame #3: <unknown function> + 0x13eef0 (0x7fa0f3cf6ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1994911Z frame #4: <unknown function> + 0x1378c4 (0x7fa0f3cef8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1995150Z frame #5: <unknown function> + 0x95b2b (0x7fa0f3c4db2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1995381Z frame #6: <unknown function> + 0x34b71 (0x7fa0f3becb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.1995605Z frame #7: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1995820Z frame #8: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1995832Z 
2025-07-17T11:07:26.1995925Z SIGABRT(6), PID: 1800823, Thread 1805149: 
2025-07-17T11:07:26.1996474Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.1996700Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1996919Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1997208Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1997595Z frame #4: <unknown function> + 0x5a02023 (0x7f39d7a09023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1998248Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f39d7a0d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1998853Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f39d7a041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.1999240Z frame #7: <unknown function> + 0x79d655 (0x7f39e8ef4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.1999542Z frame #8: <unknown function> + 0xdc253 (0x7f399ff3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.1999756Z frame #9: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1999977Z frame #10: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.1999981Z 
2025-07-17T11:07:26.2000073Z SIGABRT(6), PID: 1800818, Thread 1805092: 
2025-07-17T11:07:26.2000557Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2000769Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2000946Z frame #2: __poll + 0x4f (0x7fa277a53bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2001160Z frame #3: <unknown function> + 0xd4540 (0x7fa13be66540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2001377Z frame #4: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2001590Z frame #5: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2001597Z 
2025-07-17T11:07:26.2001693Z SIGABRT(6), PID: 1800818, Thread 1805100: 
2025-07-17T11:07:26.2002165Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2002380Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2002545Z frame #2: __poll + 0x4f (0x7fa277a53bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2002776Z frame #3: <unknown function> + 0xd65a2 (0x7fa13be685a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2002987Z frame #4: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2003203Z frame #5: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2003208Z 
2025-07-17T11:07:26.2003302Z SIGABRT(6), PID: 1800818, Thread 1805108: 
2025-07-17T11:07:26.2003779Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2003985Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2004199Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2004401Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2004696Z frame #4: <unknown function> + 0xd1e7f (0x7fa13be63e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2004908Z frame #5: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2005129Z frame #6: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2005134Z 
2025-07-17T11:07:26.2005227Z SIGABRT(6), PID: 1800818, Thread 1805123: 
2025-07-17T11:07:26.2005772Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2005984Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2006252Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2006462Z frame #3: <unknown function> + 0x9ca53 (0x7fa2779d7a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2006729Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5644aec95bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2006958Z frame #5: <unknown function> + 0x1d314f (0x5644aed5414f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2007261Z frame #6: <unknown function> + 0x1d2f6b (0x5644aed53f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2007487Z frame #7: <unknown function> + 0x6df36 (0x5644aebeef36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2007742Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2007975Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2008232Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2008465Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2008712Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2008941Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2009189Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2009416Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2009664Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2009888Z frame #17: <unknown function> + 0x147734 (0x5644aecc8734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2010125Z frame #18: <unknown function> + 0x21ca82 (0x5644aed9da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2010350Z frame #19: <unknown function> + 0x21c9e4 (0x5644aed9d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2010568Z frame #20: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2010791Z frame #21: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2010797Z 
2025-07-17T11:07:26.2010903Z SIGABRT(6), PID: 1800822, Thread 1805176: 
2025-07-17T11:07:26.2011384Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2011610Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2011816Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2012028Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2012410Z frame #4: <unknown function> + 0x5a02023 (0x7f22193e5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2013058Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f22193e91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2013674Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f22193e01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2014119Z frame #7: <unknown function> + 0x79d655 (0x7f222a8d0655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2014366Z frame #8: <unknown function> + 0xdc253 (0x7f21e191b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2014639Z frame #9: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2014868Z frame #10: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2014874Z 
2025-07-17T11:07:26.2014970Z SIGABRT(6), PID: 1800818, Thread 1805135: 
2025-07-17T11:07:26.2015456Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2015731Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2015964Z frame #2: read + 0x4c (0x7fa277a4f81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2016150Z frame #3: _Py_read + 0x4c (0x5644aed5ba6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2016381Z frame #4: <unknown function> + 0x208c3b (0x5644aed89c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2016606Z frame #5: <unknown function> + 0x14648d (0x5644aecc748d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2016861Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5644aecd6215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2017085Z frame #7: <unknown function> + 0x208ae7 (0x5644aed89ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2017310Z frame #8: <unknown function> + 0x208472 (0x5644aed89472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2017624Z frame #9: <unknown function> + 0x1da6ee (0x5644aed5b6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2017856Z frame #10: <unknown function> + 0x1483bc (0x5644aecc93bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2018102Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2018336Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2018579Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5644aecad3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2018806Z frame #14: <unknown function> + 0x147734 (0x5644aecc8734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2019055Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5644aecafc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2019291Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2019530Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2019764Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5644aecbd0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2020008Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5644aecad7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2020235Z frame #20: <unknown function> + 0x147734 (0x5644aecc8734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2020453Z frame #21: <unknown function> + 0x21ca82 (0x5644aed9da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2020679Z frame #22: <unknown function> + 0x21c9e4 (0x5644aed9d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2020981Z frame #23: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2021201Z frame #24: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2021208Z 
2025-07-17T11:07:26.2021299Z SIGABRT(6), PID: 1800823, Thread 1805150: 
2025-07-17T11:07:26.2021787Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2022073Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2022290Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2022496Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2022956Z frame #4: <unknown function> + 0x5a02023 (0x7f39d7a09023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2023529Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f39d7a0d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2024230Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f39d7a041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2024615Z frame #7: <unknown function> + 0x79d655 (0x7f39e8ef4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2024852Z frame #8: <unknown function> + 0xdc253 (0x7f399ff3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2025069Z frame #9: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2025289Z frame #10: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2025295Z 
2025-07-17T11:07:26.2025385Z SIGABRT(6), PID: 1800822, Thread 1805179: 
2025-07-17T11:07:26.2025867Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2026075Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2026297Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2026500Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2026877Z frame #4: <unknown function> + 0x5a02023 (0x7f22193e5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2027443Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f22193e91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2028045Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f22193e01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2028443Z frame #7: <unknown function> + 0x79d655 (0x7f222a8d0655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2028674Z frame #8: <unknown function> + 0xdc253 (0x7f21e191b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2028886Z frame #9: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2029100Z frame #10: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2029104Z 
2025-07-17T11:07:26.2029209Z SIGABRT(6), PID: 1800818, Thread 1805178: 
2025-07-17T11:07:26.2029678Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2029991Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2030198Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2030430Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2030867Z frame #4: <unknown function> + 0x5a02023 (0x7fa1bbdfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2031443Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa1bbe001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2032098Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa1bbdf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2032500Z frame #7: <unknown function> + 0x79d655 (0x7fa1cd2e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2032794Z frame #8: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2033010Z frame #9: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2033229Z frame #10: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2033233Z 
2025-07-17T11:07:26.2033330Z SIGABRT(6), PID: 1800823, Thread 1805151: 
2025-07-17T11:07:26.2033805Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2034022Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2034231Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2034443Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2034814Z frame #4: <unknown function> + 0x5a02023 (0x7f39d7a09023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2035399Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f39d7a0d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2035995Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f39d7a041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2036385Z frame #7: <unknown function> + 0x79d655 (0x7f39e8ef4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2036624Z frame #8: <unknown function> + 0xdc253 (0x7f399ff3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2036840Z frame #9: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2037065Z frame #10: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2037069Z 
2025-07-17T11:07:26.2037165Z SIGABRT(6), PID: 1800822, Thread 1805182: 
2025-07-17T11:07:26.2037644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2037859Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2038076Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2038355Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2038735Z frame #4: <unknown function> + 0x5a02023 (0x7f22193e5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2039310Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f22193e91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2048161Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f22193e01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2048643Z frame #7: <unknown function> + 0x79d655 (0x7f222a8d0655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2048971Z frame #8: <unknown function> + 0xdc253 (0x7f21e191b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2049208Z frame #9: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2049440Z frame #10: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2049511Z 
2025-07-17T11:07:26.2049615Z SIGABRT(6), PID: 1800818, Thread 1805181: 
2025-07-17T11:07:26.2050133Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2050357Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2050573Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2050783Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2051183Z frame #4: <unknown function> + 0x5a02023 (0x7fa1bbdfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2051756Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa1bbe001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2052374Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa1bbdf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2052763Z frame #7: <unknown function> + 0x79d655 (0x7fa1cd2e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2053012Z frame #8: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2053234Z frame #9: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2053452Z frame #10: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2053459Z 
2025-07-17T11:07:26.2053561Z SIGABRT(6), PID: 1800823, Thread 1805152: 
2025-07-17T11:07:26.2054042Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2054260Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2054469Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2054681Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2055057Z frame #4: <unknown function> + 0x5a02023 (0x7f39d7a09023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2055629Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f39d7a0d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2056410Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f39d7a041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2056807Z frame #7: <unknown function> + 0x79d655 (0x7f39e8ef4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2057122Z frame #8: <unknown function> + 0xdc253 (0x7f399ff3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2057343Z frame #9: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2057563Z frame #10: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2057567Z 
2025-07-17T11:07:26.2057737Z SIGABRT(6), PID: 1800822, Thread 1805184: 
2025-07-17T11:07:26.2058219Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2058439Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2058716Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2058938Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2059311Z frame #4: <unknown function> + 0x5a02023 (0x7f22193e5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2059881Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f22193e91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2060473Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f22193e01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2060863Z frame #7: <unknown function> + 0x79d655 (0x7f222a8d0655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2061098Z frame #8: <unknown function> + 0xdc253 (0x7f21e191b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2061311Z frame #9: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2061530Z frame #10: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2061534Z 
2025-07-17T11:07:26.2061635Z SIGABRT(6), PID: 1800818, Thread 1805186: 
2025-07-17T11:07:26.2062106Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2062322Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2062531Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2062732Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2063110Z frame #4: <unknown function> + 0x5a02023 (0x7fa1bbdfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2063684Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa1bbe001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2064280Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa1bbdf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2064656Z frame #7: <unknown function> + 0x79d655 (0x7fa1cd2e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2064975Z frame #8: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2065187Z frame #9: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2065412Z frame #10: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2065417Z 
2025-07-17T11:07:26.2065511Z SIGABRT(6), PID: 1800823, Thread 1805153: 
2025-07-17T11:07:26.2066050Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2066261Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2066527Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2066734Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2067105Z frame #4: <unknown function> + 0x5a02023 (0x7f39d7a09023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2067752Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f39d7a0d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2068351Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f39d7a041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2068730Z frame #7: <unknown function> + 0x79d655 (0x7f39e8ef4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2068968Z frame #8: <unknown function> + 0xdc253 (0x7f399ff3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2069179Z frame #9: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2069398Z frame #10: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2069404Z 
2025-07-17T11:07:26.2069511Z SIGABRT(6), PID: 1800822, Thread 1805187: 
2025-07-17T11:07:26.2069996Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2070204Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2070412Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2070612Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2070986Z frame #4: <unknown function> + 0x5a02023 (0x7f22193e5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2071542Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f22193e91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2072141Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f22193e01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2072527Z frame #7: <unknown function> + 0x79d655 (0x7f222a8d0655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2072753Z frame #8: <unknown function> + 0xdc253 (0x7f21e191b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2072971Z frame #9: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2073184Z frame #10: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2073258Z 
2025-07-17T11:07:26.2073357Z SIGABRT(6), PID: 1800818, Thread 1805189: 
2025-07-17T11:07:26.2073837Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2074048Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2074312Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2074521Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2074889Z frame #4: <unknown function> + 0x5a02023 (0x7fa1bbdfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2075514Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa1bbe001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2076113Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa1bbdf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2076565Z frame #7: <unknown function> + 0x79d655 (0x7fa1cd2e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2076794Z frame #8: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2077005Z frame #9: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2077218Z frame #10: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2077222Z 
2025-07-17T11:07:26.2077318Z SIGABRT(6), PID: 1800823, Thread 1805154: 
2025-07-17T11:07:26.2077806Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2078022Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2078227Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2078434Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2078802Z frame #4: <unknown function> + 0x5a02023 (0x7f39d7a09023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2079361Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f39d7a0d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2079951Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f39d7a041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2080337Z frame #7: <unknown function> + 0x79d655 (0x7f39e8ef4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2080570Z frame #8: <unknown function> + 0xdc253 (0x7f399ff3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2080778Z frame #9: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2080990Z frame #10: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2080998Z 
2025-07-17T11:07:26.2081089Z SIGABRT(6), PID: 1800822, Thread 1805191: 
2025-07-17T11:07:26.2081563Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2081775Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2082057Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2082267Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2082640Z frame #4: <unknown function> + 0x5a02023 (0x7f22193e5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2083254Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f22193e91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2083905Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f22193e01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2084303Z frame #7: <unknown function> + 0x79d655 (0x7f222a8d0655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2084534Z frame #8: <unknown function> + 0xdc253 (0x7f21e191b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2084746Z frame #9: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2085029Z frame #10: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2085033Z 
2025-07-17T11:07:26.2085123Z SIGABRT(6), PID: 1800818, Thread 1805190: 
2025-07-17T11:07:26.2085598Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2085806Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2086013Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2086212Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2086594Z frame #4: <unknown function> + 0x5a02023 (0x7fa1bbdfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2087152Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa1bbe001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2087746Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa1bbdf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2088120Z frame #7: <unknown function> + 0x79d655 (0x7fa1cd2e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2088368Z frame #8: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2088585Z frame #9: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2088803Z frame #10: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2088809Z 
2025-07-17T11:07:26.2088897Z SIGABRT(6), PID: 1800823, Thread 1805155: 
2025-07-17T11:07:26.2089376Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f39a1cd2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2089586Z frame #1: <unknown function> + 0x42520 (0x7f3a9358a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2089793Z frame #2: <unknown function> + 0x91117 (0x7f3a935d9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2089997Z frame #3: pthread_cond_wait + 0x211 (0x7f3a935dba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2090366Z frame #4: <unknown function> + 0x5a02023 (0x7f39d7a09023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2091000Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f39d7a0d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2091589Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f39d7a041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2092025Z frame #7: <unknown function> + 0x79d655 (0x7f39e8ef4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2092257Z frame #8: <unknown function> + 0xdc253 (0x7f399ff3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2092530Z frame #9: <unknown function> + 0x94ac3 (0x7f3a935dcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2092763Z frame #10: <unknown function> + 0x126850 (0x7f3a9366e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2092769Z 
2025-07-17T11:07:26.2092862Z SIGABRT(6), PID: 1800822, Thread 1805192: 
2025-07-17T11:07:26.2093332Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f21e36aeb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2093609Z frame #1: <unknown function> + 0x42520 (0x7f22d4f69520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2093816Z frame #2: <unknown function> + 0x91117 (0x7f22d4fb8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2094018Z frame #3: pthread_cond_wait + 0x211 (0x7f22d4fbaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2094383Z frame #4: <unknown function> + 0x5a02023 (0x7f22193e5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2094947Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f22193e91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2095537Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f22193e01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2095992Z frame #7: <unknown function> + 0x79d655 (0x7f222a8d0655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2096224Z frame #8: <unknown function> + 0xdc253 (0x7f21e191b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2096435Z frame #9: <unknown function> + 0x94ac3 (0x7f22d4fbbac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2096647Z frame #10: <unknown function> + 0x126850 (0x7f22d504d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2096651Z 
2025-07-17T11:07:26.2096746Z SIGABRT(6), PID: 1800818, Thread 1805193: 
2025-07-17T11:07:26.2097225Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2097441Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2097645Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2097848Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2098220Z frame #4: <unknown function> + 0x5a02023 (0x7fa1bbdfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2098786Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa1bbe001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2099386Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa1bbdf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2099852Z frame #7: <unknown function> + 0x79d655 (0x7fa1cd2e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2100090Z frame #8: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2100299Z frame #9: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2100587Z frame #10: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2100592Z 
2025-07-17T11:07:26.2100684Z SIGABRT(6), PID: 1800818, Thread 1805194: 
2025-07-17T11:07:26.2101247Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2101464Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2101673Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2101872Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2102318Z frame #4: <unknown function> + 0x5a02023 (0x7fa1bbdfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2102873Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa1bbe001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2103475Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa1bbdf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2103850Z frame #7: <unknown function> + 0x79d655 (0x7fa1cd2e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2104083Z frame #8: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2104292Z frame #9: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2104508Z frame #10: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2104514Z 
2025-07-17T11:07:26.2104603Z SIGABRT(6), PID: 1800818, Thread 1805197: 
2025-07-17T11:07:26.2105073Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa1860c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2105287Z frame #1: <unknown function> + 0x42520 (0x7fa27797d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2105508Z frame #2: <unknown function> + 0x91117 (0x7fa2779cc117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2105707Z frame #3: pthread_cond_wait + 0x211 (0x7fa2779cea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2106081Z frame #4: <unknown function> + 0x5a02023 (0x7fa1bbdfc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2106640Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa1bbe001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2107235Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa1bbdf71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2107626Z frame #7: <unknown function> + 0x79d655 (0x7fa1cd2e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2107859Z frame #8: <unknown function> + 0xdc253 (0x7fa184332253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2108142Z frame #9: <unknown function> + 0x94ac3 (0x7fa2779cfac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2108352Z frame #10: <unknown function> + 0x126850 (0x7fa277a61850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2108358Z 
2025-07-17T11:07:26.2108464Z ('RERUN', {'yellow': True}) [25.5489s] [100%]
2025-07-17T11:07:26.2109181Z distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph I0717 11:01:36.245000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1805788
2025-07-17T11:07:26.2109560Z I0717 11:01:36.246000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1805789
2025-07-17T11:07:26.2109909Z I0717 11:01:36.247000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1805790
2025-07-17T11:07:26.2110309Z I0717 11:01:36.248000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1805791
2025-07-17T11:07:26.2110667Z I0717 11:01:36.249000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1805792
2025-07-17T11:07:26.2111012Z I0717 11:01:36.250000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1805793
2025-07-17T11:07:26.2111415Z I0717 11:01:36.251000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1805794
2025-07-17T11:07:26.2111773Z I0717 11:01:36.252000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1805795
2025-07-17T11:07:26.2112789Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.2112884Z   import pkg_resources
2025-07-17T11:07:26.2113865Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.2113949Z   import pkg_resources
2025-07-17T11:07:26.2114917Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.2114999Z   import pkg_resources
2025-07-17T11:07:26.2115971Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.2116055Z   import pkg_resources
2025-07-17T11:07:26.2117019Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.2117099Z   import pkg_resources
2025-07-17T11:07:26.2118079Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.2118227Z   import pkg_resources
2025-07-17T11:07:26.2119196Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.2119275Z   import pkg_resources
2025-07-17T11:07:26.2120307Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.2120387Z   import pkg_resources
2025-07-17T11:07:26.2120932Z [rank1]:W0717 11:01:58.222000 1805789 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.2121407Z [rank5]:W0717 11:01:58.225000 1805793 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.2121945Z [rank0]:W0717 11:01:58.225000 1805788 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.2122410Z [rank6]:W0717 11:01:58.225000 1805794 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.2122877Z [rank7]:W0717 11:01:58.226000 1805795 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.2123341Z [rank3]:W0717 11:01:58.227000 1805791 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.2123814Z [rank2]:W0717 11:01:58.227000 1805790 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.2124283Z [rank4]:W0717 11:01:58.229000 1805792 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.2124570Z Memory access fault by GPU node-4 (Agent handle: 0x565266e5e6b0) on address 0x7f968f423000. Reason: Unknown.
2025-07-17T11:07:26.2124845Z Memory access fault by GPU node-3 (Agent handle: 0x55700e28bb10) on address 0x7f7ee7e23000. Reason: Unknown.
2025-07-17T11:07:26.2125112Z Memory access fault by GPU node-2 (Agent handle: 0x556993558930) on address 0x7f893be04000. Reason: Unknown.
2025-07-17T11:07:26.2125379Z Memory access fault by GPU node-8 (Agent handle: 0x563209e8b480) on address 0x7f6392a1f000. Reason: Unknown.
2025-07-17T11:07:26.2125645Z Memory access fault by GPU node-5 (Agent handle: 0x5561c5e43d80) on address 0x7f97ede06000. Reason: Unknown.
2025-07-17T11:07:26.2125912Z Memory access fault by GPU node-6 (Agent handle: 0x5585fd1225f0) on address 0x7f36e9e05000. Reason: Unknown.
2025-07-17T11:07:26.2126178Z Memory access fault by GPU node-9 (Agent handle: 0x5579ed44aea0) on address 0x7f861be1f000. Reason: Unknown.
2025-07-17T11:07:26.2126443Z Memory access fault by GPU node-7 (Agent handle: 0x556eda38eeb0) on address 0x7f5943e04000. Reason: Unknown.
2025-07-17T11:07:26.2126540Z SIGABRT(6), PID: 1805789, Thread 1805789: 
2025-07-17T11:07:26.2127031Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2127249Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2127490Z frame #2: <unknown function> + 0x7744b (0x7f872d52b44b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2127793Z frame #3: <unknown function> + 0x76c8e (0x7f872d52ac8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2128028Z frame #4: <unknown function> + 0x6ad69 (0x7f872d51ed69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2128249Z frame #5: <unknown function> + 0x3f54db (0x7f87be1834db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2128466Z frame #6: <unknown function> + 0x3e11d8 (0x7f87be16f1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2128867Z frame #7: <unknown function> + 0xb2f86 (0x7f87bde40f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2129088Z frame #8: <unknown function> + 0x8e501 (0x7f87bde1c501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2129472Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f87e8c237ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.2129917Z frame #10: <unknown function> + 0xd307d9 (0x7f88071767d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2130163Z frame #11: <unknown function> + 0x139902 (0x55700394e902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2130419Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x557003945a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2130729Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2130981Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x557003945a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2131205Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2131450Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2131675Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2131910Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2132138Z frame #19: <unknown function> + 0x1474b2 (0x55700395c4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2132345Z frame #20: PyObject_Call + 0xbc (0x55700395ce4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2132570Z frame #21: <unknown function> + 0x1ff69d (0x557003a1469d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2132801Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55700394a023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2133048Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x557003945cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2133283Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2133521Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5570039413bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2133744Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2133977Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5570039413bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2134211Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2134454Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x557003945a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2134711Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5570039494b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2134930Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55700395aab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2135154Z frame #32: <unknown function> + 0x205949 (0x557003a1a949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2135377Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55700394a023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2135623Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x557003945cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2135990Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2136228Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5570039413bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2136455Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2136689Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5570039413bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2137004Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2137263Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5570039413bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2137499Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2137827Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5570039423f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2138060Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2138468Z frame #44: <unknown function> + 0x7ab3f7 (0x7f8806bf13f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2138790Z frame #45: <unknown function> + 0x13bc88 (0x557003950c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2139013Z frame #46: PyObject_Call + 0x20f (0x55700395cf9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2139261Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55700394698d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2139484Z frame #48: <unknown function> + 0x147608 (0x55700395c608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2139722Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x557003943c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2139948Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2140186Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5570039413bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2140411Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2140656Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5570039423f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2140887Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2141120Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5570039413bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2141342Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2141577Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x557003943c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2141799Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2142036Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x557003943c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2142259Z frame #60: <unknown function> + 0x147608 (0x55700395c608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2142498Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x557003943c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2142723Z frame #62: <unknown function> + 0x147608 (0x55700395c608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2142729Z 
2025-07-17T11:07:26.2142827Z SIGABRT(6), PID: 1805789, Thread 1805831: 
2025-07-17T11:07:26.2143326Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2143545Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2143760Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2144053Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2144581Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2144795Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2145077Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2145084Z 
2025-07-17T11:07:26.2145180Z SIGABRT(6), PID: 1805789, Thread 1805833: 
2025-07-17T11:07:26.2145656Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2145926Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2146149Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2146349Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2146931Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2147141Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2147354Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2147359Z 
2025-07-17T11:07:26.2147451Z SIGABRT(6), PID: 1805789, Thread 1805836: 
2025-07-17T11:07:26.2147926Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2148131Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2148351Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2148552Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2149066Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2149275Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2149488Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2149492Z 
2025-07-17T11:07:26.2149582Z SIGABRT(6), PID: 1805789, Thread 1805838: 
2025-07-17T11:07:26.2150055Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2150262Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2150467Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2150665Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2151171Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2151379Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2151585Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2151590Z 
2025-07-17T11:07:26.2151692Z SIGABRT(6), PID: 1805789, Thread 1805840: 
2025-07-17T11:07:26.2152164Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2152451Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2152660Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2152874Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2153438Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2153656Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2153865Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2153927Z 
2025-07-17T11:07:26.2154028Z SIGABRT(6), PID: 1805789, Thread 1805843: 
2025-07-17T11:07:26.2154502Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2154711Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2154977Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2155185Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2155678Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2155889Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2156099Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2156105Z 
2025-07-17T11:07:26.2156202Z SIGABRT(6), PID: 1805789, Thread 1805845: 
2025-07-17T11:07:26.2156672Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2156888Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2157095Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2157297Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2157792Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2158007Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2158218Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2158224Z 
2025-07-17T11:07:26.2158321Z SIGABRT(6), PID: 1805789, Thread 1805847: 
2025-07-17T11:07:26.2158788Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2159012Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2159221Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2159425Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2159923Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2160141Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2160429Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2160439Z 
2025-07-17T11:07:26.2160530Z SIGABRT(6), PID: 1805789, Thread 1805849: 
2025-07-17T11:07:26.2161006Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2161213Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2161478Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2161683Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2162246Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2162456Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2162674Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2162680Z 
2025-07-17T11:07:26.2162770Z SIGABRT(6), PID: 1805789, Thread 1805851: 
2025-07-17T11:07:26.2163313Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2163519Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2163731Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2163929Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2164434Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2164643Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2164857Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2164863Z 
2025-07-17T11:07:26.2164954Z SIGABRT(6), PID: 1805789, Thread 1805853: 
2025-07-17T11:07:26.2165440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2165648Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2165861Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2166061Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2166560Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2166772Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2166985Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2166990Z 
2025-07-17T11:07:26.2167083Z SIGABRT(6), PID: 1805789, Thread 1805856: 
2025-07-17T11:07:26.2167556Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2167761Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2167973Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2168179Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2168768Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2168975Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2169191Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2169195Z 
2025-07-17T11:07:26.2169293Z SIGABRT(6), PID: 1805789, Thread 1805858: 
2025-07-17T11:07:26.2169838Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2170049Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2170316Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2170522Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2171028Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2171298Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2171520Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2171524Z 
2025-07-17T11:07:26.2171628Z SIGABRT(6), PID: 1805789, Thread 1805862: 
2025-07-17T11:07:26.2172100Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2172312Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2172520Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2172728Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2173224Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2173444Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2173659Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2173663Z 
2025-07-17T11:07:26.2173761Z SIGABRT(6), PID: 1805789, Thread 1805864: 
2025-07-17T11:07:26.2174228Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2174448Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2174657Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2174863Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2175362Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2175590Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2175803Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2175807Z 
2025-07-17T11:07:26.2175979Z SIGABRT(6), PID: 1805789, Thread 1805867: 
2025-07-17T11:07:26.2176450Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2176662Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2176955Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2177162Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2177666Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2177955Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2178169Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2178174Z 
2025-07-17T11:07:26.2178273Z SIGABRT(6), PID: 1805789, Thread 1805870: 
2025-07-17T11:07:26.2178817Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2179041Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2179250Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2179530Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2180033Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2180253Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2180465Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2180469Z 
2025-07-17T11:07:26.2180569Z SIGABRT(6), PID: 1805789, Thread 1805873: 
2025-07-17T11:07:26.2181039Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2181250Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2181455Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2181665Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2182165Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2182455Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2182675Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2182679Z 
2025-07-17T11:07:26.2182774Z SIGABRT(6), PID: 1805789, Thread 1805876: 
2025-07-17T11:07:26.2183249Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2183457Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2183671Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2183869Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2184367Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2184574Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2184792Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2184796Z 
2025-07-17T11:07:26.2184973Z SIGABRT(6), PID: 1805789, Thread 1805879: 
2025-07-17T11:07:26.2185450Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2185660Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2185871Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2186133Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2186643Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2186852Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2187157Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2187164Z 
2025-07-17T11:07:26.2187259Z SIGABRT(6), PID: 1805789, Thread 1805883: 
2025-07-17T11:07:26.2187742Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2188009Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2188222Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2188425Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2188926Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2189136Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2189353Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2189357Z 
2025-07-17T11:07:26.2189449Z SIGABRT(6), PID: 1805789, Thread 1805886: 
2025-07-17T11:07:26.2189927Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2190136Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2190350Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2190550Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2191051Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2191259Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2191477Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2191480Z 
2025-07-17T11:07:26.2191574Z SIGABRT(6), PID: 1805789, Thread 1805889: 
2025-07-17T11:07:26.2192047Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2192274Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2192484Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2192688Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2193193Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2193489Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2193698Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2193704Z 
2025-07-17T11:07:26.2193802Z SIGABRT(6), PID: 1805789, Thread 1805892: 
2025-07-17T11:07:26.2194270Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2194547Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2194759Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2194974Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2195529Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2195750Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2195957Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2196041Z 
2025-07-17T11:07:26.2196144Z SIGABRT(6), PID: 1805789, Thread 1805895: 
2025-07-17T11:07:26.2196615Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2196828Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2197034Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2197258Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2197755Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2197974Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2198184Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2198188Z 
2025-07-17T11:07:26.2198290Z SIGABRT(6), PID: 1805789, Thread 1805898: 
2025-07-17T11:07:26.2198758Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2198972Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2199178Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2199389Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2199893Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2200115Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2200324Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2200328Z 
2025-07-17T11:07:26.2200428Z SIGABRT(6), PID: 1805789, Thread 1805902: 
2025-07-17T11:07:26.2200898Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2201112Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2201319Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2201597Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2202102Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2202329Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2202602Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2202612Z 
2025-07-17T11:07:26.2202707Z SIGABRT(6), PID: 1805789, Thread 1805905: 
2025-07-17T11:07:26.2203184Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2203452Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2203677Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2203884Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2204405Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2204727Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2204952Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2204956Z 
2025-07-17T11:07:26.2205046Z SIGABRT(6), PID: 1805789, Thread 1805908: 
2025-07-17T11:07:26.2205518Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2205725Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2205941Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2206139Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2206640Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2206852Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2207074Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2207078Z 
2025-07-17T11:07:26.2207170Z SIGABRT(6), PID: 1805789, Thread 1805912: 
2025-07-17T11:07:26.2207651Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2207860Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2208077Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2208277Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2208777Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2208988Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2209203Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2209207Z 
2025-07-17T11:07:26.2209301Z SIGABRT(6), PID: 1805789, Thread 1805915: 
2025-07-17T11:07:26.2209792Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2210074Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2210289Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2210491Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2211051Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2213641Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2213878Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2213884Z 
2025-07-17T11:07:26.2214074Z SIGABRT(6), PID: 1805789, Thread 1805918: 
2025-07-17T11:07:26.2214617Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2214843Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2215134Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2215344Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2215874Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2216198Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2216421Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2216425Z 
2025-07-17T11:07:26.2216526Z SIGABRT(6), PID: 1805789, Thread 1805921: 
2025-07-17T11:07:26.2217011Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2217225Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2217450Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2217660Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2218165Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2218378Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2218605Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2218610Z 
2025-07-17T11:07:26.2218703Z SIGABRT(6), PID: 1805789, Thread 1805923: 
2025-07-17T11:07:26.2219180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2219390Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2219604Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2219812Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2220321Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2220531Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2220751Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2220755Z 
2025-07-17T11:07:26.2220854Z SIGABRT(6), PID: 1805789, Thread 1805927: 
2025-07-17T11:07:26.2221328Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2221549Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2221848Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2222070Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2222814Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2223035Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2223245Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2223249Z 
2025-07-17T11:07:26.2223347Z SIGABRT(6), PID: 1805789, Thread 1805930: 
2025-07-17T11:07:26.2223891Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2224106Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2224313Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2224517Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2225017Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2225237Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2225444Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2225449Z 
2025-07-17T11:07:26.2225548Z SIGABRT(6), PID: 1805789, Thread 1805932: 
2025-07-17T11:07:26.2226011Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2226230Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2226440Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2226643Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2227136Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2227388Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2227601Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2227607Z 
2025-07-17T11:07:26.2227709Z SIGABRT(6), PID: 1805789, Thread 1805934: 
2025-07-17T11:07:26.2228176Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2228387Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2228598Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2228802Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2229295Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2229506Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2229718Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2229721Z 
2025-07-17T11:07:26.2229824Z SIGABRT(6), PID: 1805789, Thread 1805936: 
2025-07-17T11:07:26.2230358Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2230667Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2230879Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2231142Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2231639Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2231855Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2232166Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2232170Z 
2025-07-17T11:07:26.2232269Z SIGABRT(6), PID: 1805789, Thread 1805939: 
2025-07-17T11:07:26.2232751Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2232960Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2233181Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2233383Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2233885Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2234094Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2234315Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2234321Z 
2025-07-17T11:07:26.2234414Z SIGABRT(6), PID: 1805789, Thread 1805941: 
2025-07-17T11:07:26.2234897Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2235106Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2235316Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2235512Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2236011Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2236220Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2236437Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2236441Z 
2025-07-17T11:07:26.2236535Z SIGABRT(6), PID: 1805789, Thread 1805942: 
2025-07-17T11:07:26.2237021Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2237232Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2237449Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2237646Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2238140Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2238417Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2238640Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2238708Z 
2025-07-17T11:07:26.2238804Z SIGABRT(6), PID: 1805789, Thread 1805944: 
2025-07-17T11:07:26.2239350Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2239565Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2239782Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2239982Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2240549Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2240762Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2240979Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2240984Z 
2025-07-17T11:07:26.2241077Z SIGABRT(6), PID: 1805789, Thread 1805946: 
2025-07-17T11:07:26.2241553Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2241770Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2241977Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2242181Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2242678Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2242890Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2243099Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2243103Z 
2025-07-17T11:07:26.2243212Z SIGABRT(6), PID: 1805789, Thread 1805948: 
2025-07-17T11:07:26.2243681Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2243901Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2244123Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2244331Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2244842Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2245061Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2245272Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2245278Z 
2025-07-17T11:07:26.2245379Z SIGABRT(6), PID: 1805789, Thread 1805952: 
2025-07-17T11:07:26.2245847Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2246060Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2246270Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2246481Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2247053Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2247334Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2247600Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2247605Z 
2025-07-17T11:07:26.2247707Z SIGABRT(6), PID: 1805789, Thread 1805955: 
2025-07-17T11:07:26.2248187Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2248474Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2248683Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2248905Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2249429Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2249647Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2249862Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2249866Z 
2025-07-17T11:07:26.2249962Z SIGABRT(6), PID: 1805789, Thread 1805958: 
2025-07-17T11:07:26.2250440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2250656Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2250862Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2251070Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2251573Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2251801Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2252024Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2252028Z 
2025-07-17T11:07:26.2252122Z SIGABRT(6), PID: 1805789, Thread 1805960: 
2025-07-17T11:07:26.2252602Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2252810Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2253025Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2253227Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2253733Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2253942Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2254194Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2254198Z 
2025-07-17T11:07:26.2254299Z SIGABRT(6), PID: 1805789, Thread 1805964: 
2025-07-17T11:07:26.2254781Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2255070Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2255295Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2255559Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2256224Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2256446Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2256678Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2256892Z 
2025-07-17T11:07:26.2256994Z SIGABRT(6), PID: 1805789, Thread 1805969: 
2025-07-17T11:07:26.2257477Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2257697Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2257910Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2258112Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2258625Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2258837Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2259059Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2259063Z 
2025-07-17T11:07:26.2259155Z SIGABRT(6), PID: 1805789, Thread 1805975: 
2025-07-17T11:07:26.2259640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2259849Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2260064Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2260268Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2260776Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2260984Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2261245Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2261249Z 
2025-07-17T11:07:26.2261344Z SIGABRT(6), PID: 1805789, Thread 1805980: 
2025-07-17T11:07:26.2261828Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2262042Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2262252Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2262453Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2262958Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2263176Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2263385Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2263389Z 
2025-07-17T11:07:26.2263595Z SIGABRT(6), PID: 1805789, Thread 1805985: 
2025-07-17T11:07:26.2264100Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2264471Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2264685Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2264897Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2265411Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2265700Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2265915Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2265920Z 
2025-07-17T11:07:26.2266016Z SIGABRT(6), PID: 1805789, Thread 1805990: 
2025-07-17T11:07:26.2266530Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2266762Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2266973Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2267185Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2267700Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2267927Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2268142Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2268148Z 
2025-07-17T11:07:26.2268249Z SIGABRT(6), PID: 1805789, Thread 1805995: 
2025-07-17T11:07:26.2268732Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2268951Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2269157Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2269370Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2269875Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2270092Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2270308Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2270313Z 
2025-07-17T11:07:26.2270415Z SIGABRT(6), PID: 1805789, Thread 1806000: 
2025-07-17T11:07:26.2270891Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2271114Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2271324Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2271533Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2272107Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2272335Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2272610Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2272620Z 
2025-07-17T11:07:26.2272713Z SIGABRT(6), PID: 1805789, Thread 1806005: 
2025-07-17T11:07:26.2273264Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2273494Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2273712Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2273982Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2274486Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2274701Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2274923Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2274929Z 
2025-07-17T11:07:26.2275020Z SIGABRT(6), PID: 1805789, Thread 1806011: 
2025-07-17T11:07:26.2275491Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2275698Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2275916Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2276121Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2276623Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2276836Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2277052Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2277056Z 
2025-07-17T11:07:26.2277148Z SIGABRT(6), PID: 1805789, Thread 1806016: 
2025-07-17T11:07:26.2277627Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2277839Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2278053Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2278274Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2278782Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2278995Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2279217Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2279222Z 
2025-07-17T11:07:26.2279314Z SIGABRT(6), PID: 1805789, Thread 1806021: 
2025-07-17T11:07:26.2279795Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2280003Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2280288Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2280492Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2281063Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2281327Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2281547Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2281551Z 
2025-07-17T11:07:26.2281643Z SIGABRT(6), PID: 1805789, Thread 1806026: 
2025-07-17T11:07:26.2282120Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2282390Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2282604Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2282809Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2283317Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2283528Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2283747Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2283751Z 
2025-07-17T11:07:26.2283842Z SIGABRT(6), PID: 1805789, Thread 1806031: 
2025-07-17T11:07:26.2284326Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2284545Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2284753Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2284962Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2285471Z frame #4: <unknown function> + 0x3475ab (0x7f88ae1655ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2285691Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2285903Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2285909Z 
2025-07-17T11:07:26.2286005Z SIGABRT(6), PID: 1805789, Thread 1806858: 
2025-07-17T11:07:26.2286469Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2286932Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f87bf9c208a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2287145Z frame #2: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2287347Z frame #3: pthread_kill + 0x12c (0x7f88b12cf9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2287516Z frame #4: raise + 0x16 (0x7f88b127b476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2287686Z frame #5: abort + 0xd3 (0x7f88b12617f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2287935Z frame #6: <unknown function> + 0x972cc (0x7f872d54b2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2288182Z frame #7: <unknown function> + 0x95f47 (0x7f872d549f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2288412Z frame #8: <unknown function> + 0x34b71 (0x7f872d4e8b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2288697Z frame #9: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2288981Z frame #10: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2288986Z 
2025-07-17T11:07:26.2289084Z SIGABRT(6), PID: 1805789, Thread 1809976: 
2025-07-17T11:07:26.2289625Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2289844Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2290012Z frame #2: __poll + 0x4f (0x7f88b1351bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2290535Z frame #3: <unknown function> + 0x489f (0x7f88b0e5d89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.2290767Z frame #4: <unknown function> + 0x1483bc (0x55700395d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2291018Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2291248Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2291495Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2291719Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2291976Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x557003945a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2292207Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2292462Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x557003943c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2292693Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2292939Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2293163Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2293407Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2293632Z frame #16: <unknown function> + 0x147734 (0x55700395c734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2293859Z frame #17: <unknown function> + 0x21ca82 (0x557003a31a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2294081Z frame #18: <unknown function> + 0x21c9e4 (0x557003a319e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2294306Z frame #19: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2294519Z frame #20: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2294525Z 
2025-07-17T11:07:26.2294625Z SIGABRT(6), PID: 1805789, Thread 1809979: 
2025-07-17T11:07:26.2295105Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2295328Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2295536Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2295764Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f88b12cd2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2296308Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f87eb70a048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.2296749Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f87eb70cbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.2297076Z frame #6: <unknown function> + 0xdc253 (0x7f87bdc2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2297378Z frame #7: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2297597Z frame #8: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2297670Z 
2025-07-17T11:07:26.2297774Z SIGABRT(6), PID: 1805789, Thread 1809980: 
2025-07-17T11:07:26.2298253Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2298469Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2298751Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2298985Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f88b12cd2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2299464Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f87eb704741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.2299704Z frame #5: <unknown function> + 0xdc253 (0x7f87bdc2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2299920Z frame #6: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2300142Z frame #7: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2300146Z 
2025-07-17T11:07:26.2300239Z SIGABRT(6), PID: 1805789, Thread 1810025: 
2025-07-17T11:07:26.2300717Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2300943Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2301120Z frame #2: ioctl + 0x3f (0x7f88b135394f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2301358Z frame #3: <unknown function> + 0x13eef0 (0x7f872d5f2ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2301597Z frame #4: <unknown function> + 0x1378c4 (0x7f872d5eb8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2301832Z frame #5: <unknown function> + 0x95b2b (0x7f872d549b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2302064Z frame #6: <unknown function> + 0x34b71 (0x7f872d4e8b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2302285Z frame #7: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2302504Z frame #8: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2302510Z 
2025-07-17T11:07:26.2302607Z SIGABRT(6), PID: 1805789, Thread 1810605: 
2025-07-17T11:07:26.2303077Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2303294Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2303466Z frame #2: __poll + 0x4f (0x7f88b1351bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2303685Z frame #3: <unknown function> + 0xd4540 (0x7f8775762540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2303895Z frame #4: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2304110Z frame #5: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2304114Z 
2025-07-17T11:07:26.2304206Z SIGABRT(6), PID: 1805789, Thread 1810613: 
2025-07-17T11:07:26.2304689Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2304986Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2305163Z frame #2: __poll + 0x4f (0x7f88b1351bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2305444Z frame #3: <unknown function> + 0xd65a2 (0x7f87757645a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2305724Z frame #4: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2305942Z frame #5: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2305946Z 
2025-07-17T11:07:26.2306044Z SIGABRT(6), PID: 1805789, Thread 1810618: 
2025-07-17T11:07:26.2306510Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2306788Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2307001Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2307220Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2307444Z frame #4: <unknown function> + 0xd1e7f (0x7f877575fe7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2307660Z frame #5: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2307873Z frame #6: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2307878Z 
2025-07-17T11:07:26.2307977Z SIGABRT(6), PID: 1805789, Thread 1810622: 
2025-07-17T11:07:26.2308443Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2308664Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2308875Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2309093Z frame #3: <unknown function> + 0x9ca53 (0x7f88b12d5a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2309364Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x557003929bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2309608Z frame #5: <unknown function> + 0x1d314f (0x5570039e814f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2309830Z frame #6: <unknown function> + 0x1d2f6b (0x5570039e7f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2310058Z frame #7: <unknown function> + 0x6df36 (0x557003882f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2310302Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2310537Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2310777Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2311013Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2311255Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2311497Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2311736Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2311968Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2312206Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2312442Z frame #17: <unknown function> + 0x147734 (0x55700395c734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2312665Z frame #18: <unknown function> + 0x21ca82 (0x557003a31a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2312962Z frame #19: <unknown function> + 0x21c9e4 (0x557003a319e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2313237Z frame #20: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2313459Z frame #21: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2313520Z 
2025-07-17T11:07:26.2313618Z SIGABRT(6), PID: 1805789, Thread 1810631: 
2025-07-17T11:07:26.2314109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2314327Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2314569Z frame #2: read + 0x4c (0x7f88b134d81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2314757Z frame #3: _Py_read + 0x4c (0x5570039efa6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2314994Z frame #4: <unknown function> + 0x208c3b (0x557003a1dc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2315218Z frame #5: <unknown function> + 0x14648d (0x55700395b48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2315473Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55700396a215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2315700Z frame #7: <unknown function> + 0x208ae7 (0x557003a1dae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2315925Z frame #8: <unknown function> + 0x208472 (0x557003a1d472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2316151Z frame #9: <unknown function> + 0x1da6ee (0x5570039ef6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2316384Z frame #10: <unknown function> + 0x1483bc (0x55700395d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2316635Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2316874Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2317110Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5570039413bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2317338Z frame #14: <unknown function> + 0x147734 (0x55700395c734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2317590Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x557003943c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2317821Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2318057Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2318289Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5570039510ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2318522Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5570039417b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2318752Z frame #20: <unknown function> + 0x147734 (0x55700395c734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2318977Z frame #21: <unknown function> + 0x21ca82 (0x557003a31a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2319205Z frame #22: <unknown function> + 0x21c9e4 (0x557003a319e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2319424Z frame #23: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2319641Z frame #24: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2319645Z 
2025-07-17T11:07:26.2319737Z SIGABRT(6), PID: 1805789, Thread 1810646: 
2025-07-17T11:07:26.2320223Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2320441Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2320732Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2320943Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2321414Z frame #4: <unknown function> + 0x5a02023 (0x7f87f56f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2322076Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87f56fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2322677Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87f56f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2323130Z frame #7: <unknown function> + 0x79d655 (0x7f8806be3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2323375Z frame #8: <unknown function> + 0xdc253 (0x7f87bdc2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2323588Z frame #9: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2323814Z frame #10: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2323818Z 
2025-07-17T11:07:26.2323918Z SIGABRT(6), PID: 1805788, Thread 1805788: 
2025-07-17T11:07:26.2324410Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2324627Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2324857Z frame #2: <unknown function> + 0x77435 (0x7f9180e33435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2325094Z frame #3: <unknown function> + 0x76c8e (0x7f9180e32c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2325323Z frame #4: <unknown function> + 0x6ad69 (0x7f9180e26d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2325547Z frame #5: <unknown function> + 0x3f54db (0x7f9211a8b4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2325761Z frame #6: <unknown function> + 0x3e11d8 (0x7f9211a771d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2325999Z frame #7: <unknown function> + 0xb2f86 (0x7f9211748f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2326212Z frame #8: <unknown function> + 0x8e501 (0x7f9211724501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2326611Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f923c52b7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.2327006Z frame #10: <unknown function> + 0xd307d9 (0x7f925aa7e7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2327250Z frame #11: <unknown function> + 0x139902 (0x556988e11902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2327502Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x556988e08a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2327738Z frame #13: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2327984Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x556988e08a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2328216Z frame #15: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2328457Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2328691Z frame #17: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2328928Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2329233Z frame #19: <unknown function> + 0x1474b2 (0x556988e1f4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2329447Z frame #20: PyObject_Call + 0xbc (0x556988e1fe4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2329746Z frame #21: <unknown function> + 0x1ff69d (0x556988ed769d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2330032Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x556988e0d023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2330284Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x556988e08cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2330512Z frame #24: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2330764Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x556988e043bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2331062Z frame #26: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2331308Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x556988e043bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2331538Z frame #28: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2331787Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x556988e08a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2332045Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x556988e0c4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2332270Z frame #31: _PyObject_Call_Prepend + 0x69 (0x556988e1dab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2332496Z frame #32: <unknown function> + 0x205949 (0x556988edd949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2332730Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x556988e0d023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2332995Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x556988e08cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2333236Z frame #35: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2333476Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x556988e043bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2333706Z frame #37: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2333940Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x556988e043bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2334169Z frame #39: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2334404Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x556988e043bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2334633Z frame #41: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2334874Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x556988e053f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2335103Z frame #43: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2335494Z frame #44: <unknown function> + 0x7ab3f7 (0x7f925a4f93f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2335728Z frame #45: <unknown function> + 0x13bc88 (0x556988e13c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2336010Z frame #46: PyObject_Call + 0x20f (0x556988e1ff9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2336264Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x556988e0998d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2336484Z frame #48: <unknown function> + 0x147608 (0x556988e1f608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2336732Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x556988e06c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2336958Z frame #50: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2337199Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x556988e043bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2337510Z frame #52: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2337830Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x556988e053f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2338055Z frame #54: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2338386Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x556988e043bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2338616Z frame #56: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2338860Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x556988e06c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2339159Z frame #58: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2339401Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x556988e06c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2339624Z frame #60: <unknown function> + 0x147608 (0x556988e1f608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2339869Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x556988e06c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2340088Z frame #62: <unknown function> + 0x147608 (0x556988e1f608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2340095Z 
2025-07-17T11:07:26.2340194Z SIGABRT(6), PID: 1805789, Thread 1810647: 
2025-07-17T11:07:26.2340669Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2340897Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2341111Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2341317Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2341694Z frame #4: <unknown function> + 0x5a02023 (0x7f87f56f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2342271Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87f56fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2342872Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87f56f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2343270Z frame #7: <unknown function> + 0x79d655 (0x7f8806be3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2343510Z frame #8: <unknown function> + 0xdc253 (0x7f87bdc2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2343727Z frame #9: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2343945Z frame #10: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2343951Z 
2025-07-17T11:07:26.2344052Z SIGABRT(6), PID: 1805788, Thread 1805806: 
2025-07-17T11:07:26.2344526Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2344742Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2344948Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2345154Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2345677Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2345975Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2346255Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2346259Z 
2025-07-17T11:07:26.2346358Z SIGABRT(6), PID: 1805788, Thread 1805807: 
2025-07-17T11:07:26.2346889Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2347105Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2347313Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2347580Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2348090Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2348309Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2348530Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2348542Z 
2025-07-17T11:07:26.2348637Z SIGABRT(6), PID: 1805788, Thread 1805808: 
2025-07-17T11:07:26.2349116Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2349326Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2349540Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2349739Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2350243Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2350463Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2350690Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2350694Z 
2025-07-17T11:07:26.2350790Z SIGABRT(6), PID: 1805788, Thread 1805809: 
2025-07-17T11:07:26.2351280Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2351487Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2351699Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2351902Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2352402Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2352616Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2352841Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2352845Z 
2025-07-17T11:07:26.2352938Z SIGABRT(6), PID: 1805788, Thread 1805810: 
2025-07-17T11:07:26.2353430Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2353640Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2353853Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2354133Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2354726Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2354997Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2355215Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2355219Z 
2025-07-17T11:07:26.2355314Z SIGABRT(6), PID: 1805789, Thread 1810648: 
2025-07-17T11:07:26.2355813Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2356095Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2356310Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2356512Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2356892Z frame #4: <unknown function> + 0x5a02023 (0x7f87f56f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2357463Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87f56fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2358068Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87f56f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2358456Z frame #7: <unknown function> + 0x79d655 (0x7f8806be3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2358696Z frame #8: <unknown function> + 0xdc253 (0x7f87bdc2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2358906Z frame #9: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2359126Z frame #10: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2359130Z 
2025-07-17T11:07:26.2359224Z SIGABRT(6), PID: 1805788, Thread 1805811: 
2025-07-17T11:07:26.2359699Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2359916Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2360123Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2360331Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2360861Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2361090Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2361305Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2361309Z 
2025-07-17T11:07:26.2361405Z SIGABRT(6), PID: 1805788, Thread 1805812: 
2025-07-17T11:07:26.2361868Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2362082Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2362289Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2362560Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2363065Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2363340Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2363613Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2363618Z 
2025-07-17T11:07:26.2363716Z SIGABRT(6), PID: 1805788, Thread 1805813: 
2025-07-17T11:07:26.2364182Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2364463Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2364668Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2364871Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2365370Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2365586Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2365794Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2365799Z 
2025-07-17T11:07:26.2365899Z SIGABRT(6), PID: 1805788, Thread 1805814: 
2025-07-17T11:07:26.2366369Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2366582Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2366788Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2366992Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2367497Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2367718Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2367927Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2368024Z SIGABRT(6), PID: 1805789, Thread 1810649: 
2025-07-17T11:07:26.2368498Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2368720Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2368930Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2369138Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2369519Z frame #4: <unknown function> + 0x5a02023 (0x7f87f56f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2370102Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87f56fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2370708Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87f56f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2371163Z frame #7: <unknown function> + 0x79d655 (0x7f8806be3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2371408Z frame #8: <unknown function> + 0xdc253 (0x7f87bdc2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2371682Z frame #9: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2371965Z frame #10: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2371970Z 
2025-07-17T11:07:26.2371973Z 
2025-07-17T11:07:26.2372071Z SIGABRT(6), PID: 1805788, Thread 1805815: 
2025-07-17T11:07:26.2372564Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2372841Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2373055Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2373258Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2373757Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2373972Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2374186Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2374191Z 
2025-07-17T11:07:26.2374282Z SIGABRT(6), PID: 1805788, Thread 1805816: 
2025-07-17T11:07:26.2374762Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2374978Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2375192Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2375391Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2375890Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2376174Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2376392Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2376396Z 
2025-07-17T11:07:26.2376488Z SIGABRT(6), PID: 1805788, Thread 1805817: 
2025-07-17T11:07:26.2376959Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2377168Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2377389Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2377593Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2378100Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2378312Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2378528Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2378532Z 
2025-07-17T11:07:26.2378625Z SIGABRT(6), PID: 1805788, Thread 1805818: 
2025-07-17T11:07:26.2379103Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2379405Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2379630Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2379907Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2380480Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2380701Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2380911Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2381005Z 
2025-07-17T11:07:26.2381105Z SIGABRT(6), PID: 1805789, Thread 1810650: 
2025-07-17T11:07:26.2381577Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2381793Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2382003Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2382212Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2382588Z frame #4: <unknown function> + 0x5a02023 (0x7f87f56f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2383158Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87f56fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2383753Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87f56f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2384140Z frame #7: <unknown function> + 0x79d655 (0x7f8806be3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2384373Z frame #8: <unknown function> + 0xdc253 (0x7f87bdc2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2384602Z frame #9: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2384823Z frame #10: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2384827Z 
2025-07-17T11:07:26.2384921Z SIGABRT(6), PID: 1805788, Thread 1805819: 
2025-07-17T11:07:26.2385392Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2385606Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2385813Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2386017Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2386519Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2386735Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2386946Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2386950Z 
2025-07-17T11:07:26.2387051Z SIGABRT(6), PID: 1805788, Thread 1805820: 
2025-07-17T11:07:26.2387525Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2387829Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2388043Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2388313Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2388994Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2389213Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2389440Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2389444Z 
2025-07-17T11:07:26.2389616Z SIGABRT(6), PID: 1805788, Thread 1805821: 
2025-07-17T11:07:26.2390092Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2390300Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2390510Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2390706Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2391206Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2391413Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2391633Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2391639Z 
2025-07-17T11:07:26.2391737Z SIGABRT(6), PID: 1805788, Thread 1805822: 
2025-07-17T11:07:26.2392209Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2392414Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2392624Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2392824Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2393327Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2393533Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2393747Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2393751Z 
2025-07-17T11:07:26.2393842Z SIGABRT(6), PID: 1805788, Thread 1805823: 
2025-07-17T11:07:26.2394333Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2394542Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2394751Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2394948Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2395447Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2395654Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2395868Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2395872Z 
2025-07-17T11:07:26.2395963Z SIGABRT(6), PID: 1805789, Thread 1810651: 
2025-07-17T11:07:26.2396520Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2396793Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2397076Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2397276Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2397654Z frame #4: <unknown function> + 0x5a02023 (0x7f87f56f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2398286Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87f56fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2398890Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87f56f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2399278Z frame #7: <unknown function> + 0x79d655 (0x7f8806be3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2399517Z frame #8: <unknown function> + 0xdc253 (0x7f87bdc2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2399727Z frame #9: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2399945Z frame #10: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2399951Z 
2025-07-17T11:07:26.2400045Z SIGABRT(6), PID: 1805788, Thread 1805824: 
2025-07-17T11:07:26.2400530Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2400745Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2400952Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2401176Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2401678Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2401891Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2402100Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2402108Z 
2025-07-17T11:07:26.2402203Z SIGABRT(6), PID: 1805788, Thread 1805825: 
2025-07-17T11:07:26.2402674Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2402888Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2403096Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2403300Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2403803Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2404018Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2404230Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2404234Z 
2025-07-17T11:07:26.2404329Z SIGABRT(6), PID: 1805788, Thread 1805826: 
2025-07-17T11:07:26.2404860Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2405132Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2405406Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2405613Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2406120Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2406405Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2406618Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2406622Z 
2025-07-17T11:07:26.2406722Z SIGABRT(6), PID: 1805788, Thread 1805827: 
2025-07-17T11:07:26.2407193Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2407406Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2407613Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2407814Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2408310Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2408534Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2408748Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2408753Z 
2025-07-17T11:07:26.2408849Z SIGABRT(6), PID: 1805788, Thread 1805828: 
2025-07-17T11:07:26.2409319Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2409531Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2409737Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2409951Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2410453Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2410671Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2410883Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2410897Z 
2025-07-17T11:07:26.2410992Z SIGABRT(6), PID: 1805789, Thread 1810652: 
2025-07-17T11:07:26.2411477Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2411688Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2411899Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2412095Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2412484Z frame #4: <unknown function> + 0x5a02023 (0x7f87f56f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2413126Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87f56fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2413788Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87f56f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2414231Z frame #7: <unknown function> + 0x79d655 (0x7f8806be3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2414474Z frame #8: <unknown function> + 0xdc253 (0x7f87bdc2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2414689Z frame #9: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2414989Z frame #10: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2414994Z 
2025-07-17T11:07:26.2415087Z SIGABRT(6), PID: 1805788, Thread 1805829: 
2025-07-17T11:07:26.2415575Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2415793Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2416079Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2416288Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2416808Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2417022Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2417235Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2417239Z 
2025-07-17T11:07:26.2417330Z SIGABRT(6), PID: 1805788, Thread 1805830: 
2025-07-17T11:07:26.2417807Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2418035Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2418256Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2418457Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2418958Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2419168Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2419384Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2419388Z 
2025-07-17T11:07:26.2419480Z SIGABRT(6), PID: 1805788, Thread 1805832: 
2025-07-17T11:07:26.2419959Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2420167Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2420383Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2420587Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2421086Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2421297Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2421601Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2421670Z 
2025-07-17T11:07:26.2421770Z SIGABRT(6), PID: 1805788, Thread 1805834: 
2025-07-17T11:07:26.2422320Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2422538Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2422747Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2422974Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2423550Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2423772Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2423984Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2423991Z 
2025-07-17T11:07:26.2424085Z SIGABRT(6), PID: 1805788, Thread 1805835: 
2025-07-17T11:07:26.2424556Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2424772Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2424979Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2425189Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2425692Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2425904Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2426114Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2426118Z 
2025-07-17T11:07:26.2426214Z SIGABRT(6), PID: 1805789, Thread 1810653: 
2025-07-17T11:07:26.2426683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f87bf9c1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2434960Z frame #1: <unknown function> + 0x42520 (0x7f88b127b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2435250Z frame #2: <unknown function> + 0x91117 (0x7f88b12ca117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2435485Z frame #3: pthread_cond_wait + 0x211 (0x7f88b12cca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2435902Z frame #4: <unknown function> + 0x5a02023 (0x7f87f56f8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2436505Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f87f56fc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2437136Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f87f56f31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2437534Z frame #7: <unknown function> + 0x79d655 (0x7f8806be3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2437782Z frame #8: <unknown function> + 0xdc253 (0x7f87bdc2e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2438012Z frame #9: <unknown function> + 0x94ac3 (0x7f88b12cdac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2438354Z frame #10: <unknown function> + 0x126850 (0x7f88b135f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2438361Z 
2025-07-17T11:07:26.2438532Z SIGABRT(6), PID: 1805788, Thread 1805837: 
2025-07-17T11:07:26.2439026Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2439310Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2439526Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2439741Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2440331Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2440553Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2440766Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2440777Z 
2025-07-17T11:07:26.2440873Z SIGABRT(6), PID: 1805788, Thread 1805839: 
2025-07-17T11:07:26.2441364Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2441574Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2441781Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2441982Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2442497Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2442710Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2442927Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2442932Z 
2025-07-17T11:07:26.2443023Z SIGABRT(6), PID: 1805788, Thread 1805841: 
2025-07-17T11:07:26.2443499Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2443705Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2443912Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2444114Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2444613Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2444823Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2445039Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2445043Z 
2025-07-17T11:07:26.2445138Z SIGABRT(6), PID: 1805788, Thread 1805842: 
2025-07-17T11:07:26.2445612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2445819Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2446027Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2446226Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2446799Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2447068Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2447295Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2447358Z 
2025-07-17T11:07:26.2447455Z SIGABRT(6), PID: 1805788, Thread 1805844: 
2025-07-17T11:07:26.2447934Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2448145Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2448422Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2448620Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2449122Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2449328Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2449542Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2449548Z 
2025-07-17T11:07:26.2449637Z SIGABRT(6), PID: 1805788, Thread 1805846: 
2025-07-17T11:07:26.2450113Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2450322Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2450528Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2450728Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2451226Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2451440Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2451646Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2451650Z 
2025-07-17T11:07:26.2451746Z SIGABRT(6), PID: 1805788, Thread 1805848: 
2025-07-17T11:07:26.2452221Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2452433Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2452647Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2452848Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2453343Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2453559Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2453766Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2453770Z 
2025-07-17T11:07:26.2453862Z SIGABRT(6), PID: 1805788, Thread 1805850: 
2025-07-17T11:07:26.2454327Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2454538Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2454814Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2455080Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2455660Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2455876Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2457204Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2457304Z 
2025-07-17T11:07:26.2457689Z SIGABRT(6), PID: 1805788, Thread 1805852: 
2025-07-17T11:07:26.2459196Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2459641Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2460041Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2460438Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2461404Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2461771Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2461993Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2462001Z 
2025-07-17T11:07:26.2462121Z SIGABRT(6), PID: 1805788, Thread 1805854: 
2025-07-17T11:07:26.2462629Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2462862Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2463079Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2463303Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2463821Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2464043Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2464259Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2464265Z 
2025-07-17T11:07:26.2464374Z SIGABRT(6), PID: 1805788, Thread 1805857: 
2025-07-17T11:07:26.2464855Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2465084Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2465305Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2465516Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2466030Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2466245Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2466467Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2466471Z 
2025-07-17T11:07:26.2466567Z SIGABRT(6), PID: 1805788, Thread 1805860: 
2025-07-17T11:07:26.2467172Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2467476Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2467798Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2468005Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2468521Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2468798Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2469021Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2469026Z 
2025-07-17T11:07:26.2469126Z SIGABRT(6), PID: 1805788, Thread 1805861: 
2025-07-17T11:07:26.2469621Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2469836Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2470050Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2470252Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2470755Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2470968Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2471186Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2471190Z 
2025-07-17T11:07:26.2471283Z SIGABRT(6), PID: 1805788, Thread 1805865: 
2025-07-17T11:07:26.2471769Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2471982Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2472196Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2472399Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2472902Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2473120Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2473342Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2473349Z 
2025-07-17T11:07:26.2473442Z SIGABRT(6), PID: 1805788, Thread 1805869: 
2025-07-17T11:07:26.2473920Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2474131Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2474367Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2474572Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2475082Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2475368Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2475592Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2475655Z 
2025-07-17T11:07:26.2475752Z SIGABRT(6), PID: 1805788, Thread 1805872: 
2025-07-17T11:07:26.2476285Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2476505Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2476718Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2476993Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2477495Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2477716Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2477927Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2477931Z 
2025-07-17T11:07:26.2478031Z SIGABRT(6), PID: 1805788, Thread 1805875: 
2025-07-17T11:07:26.2478503Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2478728Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2478942Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2479159Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2479669Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2479893Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2480103Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2480107Z 
2025-07-17T11:07:26.2480212Z SIGABRT(6), PID: 1805790, Thread 1805790: 
2025-07-17T11:07:26.2480683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2480903Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2481144Z frame #2: <unknown function> + 0x77435 (0x7f9ecb343435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2481403Z frame #3: <unknown function> + 0x76c8e (0x7f9ecb342c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2481639Z frame #4: <unknown function> + 0x6ad69 (0x7f9ecb336d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2481875Z frame #5: <unknown function> + 0x3f54db (0x7f9f5bf9b4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2482092Z frame #6: <unknown function> + 0x3e11d8 (0x7f9f5bf871d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2482328Z frame #7: <unknown function> + 0xb2f86 (0x7f9f5bc58f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2482539Z frame #8: <unknown function> + 0x8e501 (0x7f9f5bc34501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2482940Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f9f86a3b7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.2483338Z frame #10: <unknown function> + 0xd307d9 (0x7f9fa4f8e7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2483653Z frame #11: <unknown function> + 0x139902 (0x56525bf10902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2483925Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x56525bf07a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2484237Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2484548Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x56525bf07a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2484790Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2485036Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2485273Z frame #17: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2485576Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2485813Z frame #19: <unknown function> + 0x1474b2 (0x56525bf1e4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2486028Z frame #20: PyObject_Call + 0xbc (0x56525bf1ee4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2486284Z frame #21: <unknown function> + 0x1ff69d (0x56525bfd669d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2486520Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x56525bf0c023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2486782Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x56525bf07cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2487009Z frame #24: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2487259Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x56525bf033bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2487488Z frame #26: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2487735Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x56525bf033bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2487966Z frame #28: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2488221Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x56525bf07a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2488487Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x56525bf0b4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2488727Z frame #31: _PyObject_Call_Prepend + 0x69 (0x56525bf1cab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2488952Z frame #32: <unknown function> + 0x205949 (0x56525bfdc949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2489190Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x56525bf0c023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2489437Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x56525bf07cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2489667Z frame #35: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2489905Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x56525bf033bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2490135Z frame #37: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2490368Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x56525bf033bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2490600Z frame #39: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2490835Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x56525bf033bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2491075Z frame #41: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2491333Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x56525bf043f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2491563Z frame #43: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2492063Z frame #44: <unknown function> + 0x7ab3f7 (0x7f9fa4a093f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2492356Z frame #45: <unknown function> + 0x13bc88 (0x56525bf12c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2492627Z frame #46: PyObject_Call + 0x20f (0x56525bf1ef9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2492891Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x56525bf0898d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2493118Z frame #48: <unknown function> + 0x147608 (0x56525bf1e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2493368Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x56525bf05c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2493662Z frame #50: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2493908Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x56525bf033bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2494136Z frame #52: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2494385Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x56525bf043f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2494610Z frame #54: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2494854Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x56525bf033bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2495080Z frame #56: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2495330Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x56525bf05c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2495559Z frame #58: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2495814Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x56525bf05c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2496129Z frame #60: <unknown function> + 0x147608 (0x56525bf1e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2496381Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x56525bf05c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2496600Z frame #62: <unknown function> + 0x147608 (0x56525bf1e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2496607Z 
2025-07-17T11:07:26.2496717Z SIGABRT(6), PID: 1805788, Thread 1805878: 
2025-07-17T11:07:26.2497200Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2497431Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2497652Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2497868Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2498403Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2498629Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2498846Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2498850Z 
2025-07-17T11:07:26.2498955Z SIGABRT(6), PID: 1805790, Thread 1805855: 
2025-07-17T11:07:26.2499429Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2499649Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2499861Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2500161Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2500674Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2501049Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2501274Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2501280Z 
2025-07-17T11:07:26.2501384Z SIGABRT(6), PID: 1805788, Thread 1805881: 
2025-07-17T11:07:26.2501854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2502151Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2502362Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2502571Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2503089Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2503312Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2503531Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2503537Z 
2025-07-17T11:07:26.2503632Z SIGABRT(6), PID: 1805788, Thread 1805884: 
2025-07-17T11:07:26.2504115Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2504327Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2504542Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2504745Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2505252Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2505468Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2505691Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2505785Z SIGABRT(6), PID: 1805790, Thread 1805859: 
2025-07-17T11:07:26.2506271Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2506485Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2506703Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2506907Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2507416Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2507629Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2507855Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2507860Z 
2025-07-17T11:07:26.2507864Z 
2025-07-17T11:07:26.2507963Z SIGABRT(6), PID: 1805788, Thread 1805887: 
2025-07-17T11:07:26.2508517Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2508731Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2509011Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2509272Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2509782Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2509994Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2510314Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2510319Z 
2025-07-17T11:07:26.2510415Z SIGABRT(6), PID: 1805790, Thread 1805863: 
2025-07-17T11:07:26.2510906Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2511120Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2511347Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2511552Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2512059Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2512267Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2512489Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2512493Z 
2025-07-17T11:07:26.2512590Z SIGABRT(6), PID: 1805788, Thread 1805890: 
2025-07-17T11:07:26.2513066Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2513279Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2513486Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2513686Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2514186Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2514403Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2514612Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2514616Z 
2025-07-17T11:07:26.2514716Z SIGABRT(6), PID: 1805790, Thread 1805866: 
2025-07-17T11:07:26.2515198Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2515420Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2515627Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2515845Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2516338Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2516556Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2516969Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2517075Z SIGABRT(6), PID: 1805788, Thread 1805893: 
2025-07-17T11:07:26.2517616Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2517897Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2518114Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2518323Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2518825Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2519133Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2519346Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2519352Z 
2025-07-17T11:07:26.2519355Z 
2025-07-17T11:07:26.2519455Z SIGABRT(6), PID: 1805788, Thread 1805897: 
2025-07-17T11:07:26.2519925Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2520139Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2520344Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2520556Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2521052Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2521272Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2521479Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2521485Z 
2025-07-17T11:07:26.2521588Z SIGABRT(6), PID: 1805790, Thread 1805868: 
2025-07-17T11:07:26.2522056Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2522290Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2522498Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2522709Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2523206Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2523424Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2523640Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2523653Z 
2025-07-17T11:07:26.2523746Z SIGABRT(6), PID: 1805788, Thread 1805900: 
2025-07-17T11:07:26.2524224Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2524431Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2524646Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2524850Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2525426Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2525709Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2525931Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2525993Z 
2025-07-17T11:07:26.2526092Z SIGABRT(6), PID: 1805790, Thread 1805871: 
2025-07-17T11:07:26.2526571Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2526782Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2527061Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2527263Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2527772Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2527985Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2528205Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2528210Z 
2025-07-17T11:07:26.2528303Z SIGABRT(6), PID: 1805788, Thread 1805903: 
2025-07-17T11:07:26.2528776Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2528990Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2529207Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2529408Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2529950Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2530169Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2530388Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2530486Z SIGABRT(6), PID: 1805790, Thread 1805874: 
2025-07-17T11:07:26.2530975Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2531189Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2531413Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2531622Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2532143Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2532357Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2532581Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2532585Z 
2025-07-17T11:07:26.2532589Z 
2025-07-17T11:07:26.2532705Z SIGABRT(6), PID: 1805788, Thread 1805906: 
2025-07-17T11:07:26.2533186Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2533442Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2533744Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2534016Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2534589Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2534820Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2535043Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2535137Z SIGABRT(6), PID: 1805790, Thread 1805877: 
2025-07-17T11:07:26.2535695Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2535987Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2536200Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2536419Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2536926Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2537153Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2537368Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2537373Z 
2025-07-17T11:07:26.2537378Z 
2025-07-17T11:07:26.2537489Z SIGABRT(6), PID: 1805788, Thread 1805909: 
2025-07-17T11:07:26.2537967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2538189Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2538399Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2538613Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2539119Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2539344Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2539555Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2539562Z 
2025-07-17T11:07:26.2539669Z SIGABRT(6), PID: 1805790, Thread 1805880: 
2025-07-17T11:07:26.2540138Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2540365Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2540576Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2540791Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2541292Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2541515Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2541737Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2541741Z 
2025-07-17T11:07:26.2541849Z SIGABRT(6), PID: 1805788, Thread 1805911: 
2025-07-17T11:07:26.2542406Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2542703Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2542991Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2543203Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2543710Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2544016Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2544231Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2544235Z 
2025-07-17T11:07:26.2544340Z SIGABRT(6), PID: 1805790, Thread 1805882: 
2025-07-17T11:07:26.2544818Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2545048Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2545261Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2545474Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2545979Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2546196Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2546422Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2546426Z 
2025-07-17T11:07:26.2546526Z SIGABRT(6), PID: 1805788, Thread 1805914: 
2025-07-17T11:07:26.2547011Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2547221Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2547442Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2547647Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2548158Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2548372Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2548592Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2548598Z 
2025-07-17T11:07:26.2548695Z SIGABRT(6), PID: 1805790, Thread 1805885: 
2025-07-17T11:07:26.2549171Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2549382Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2549603Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2549805Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2550320Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2550611Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2550837Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2550901Z 
2025-07-17T11:07:26.2550997Z SIGABRT(6), PID: 1805788, Thread 1805917: 
2025-07-17T11:07:26.2551571Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2551786Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2552003Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2552203Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2552777Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2552989Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2553209Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2553213Z 
2025-07-17T11:07:26.2553307Z SIGABRT(6), PID: 1805790, Thread 1805888: 
2025-07-17T11:07:26.2553798Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2554014Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2554230Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2554433Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2554950Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2555165Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2555384Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2555388Z 
2025-07-17T11:07:26.2555486Z SIGABRT(6), PID: 1805788, Thread 1805920: 
2025-07-17T11:07:26.2555967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2556183Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2556396Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2556594Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2557104Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2557333Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2557549Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2557554Z 
2025-07-17T11:07:26.2557658Z SIGABRT(6), PID: 1805790, Thread 1805891: 
2025-07-17T11:07:26.2558128Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2558344Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2558552Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2558834Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2559337Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2559613Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2559888Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2559892Z 
2025-07-17T11:07:26.2559999Z SIGABRT(6), PID: 1805788, Thread 1805924: 
2025-07-17T11:07:26.2560470Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2560744Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2560953Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2561163Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2561663Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2561888Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2562100Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2562104Z 
2025-07-17T11:07:26.2562213Z SIGABRT(6), PID: 1805790, Thread 1805894: 
2025-07-17T11:07:26.2562692Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2562912Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2563123Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2563338Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2563843Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2564066Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2564283Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2564287Z 
2025-07-17T11:07:26.2564394Z SIGABRT(6), PID: 1805788, Thread 1805926: 
2025-07-17T11:07:26.2564864Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2565089Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2565298Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2565509Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2566007Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2566229Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2566439Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2566455Z 
2025-07-17T11:07:26.2566550Z SIGABRT(6), PID: 1805790, Thread 1805896: 
2025-07-17T11:07:26.2567027Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2567314Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2567599Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2567802Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2568369Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2568583Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2568804Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2568869Z 
2025-07-17T11:07:26.2568966Z SIGABRT(6), PID: 1805788, Thread 1805929: 
2025-07-17T11:07:26.2569454Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2569665Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2569880Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2570084Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2570590Z frame #4: <unknown function> + 0x3475ab (0x7f9301a6b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2570799Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2571022Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2571026Z 
2025-07-17T11:07:26.2571123Z SIGABRT(6), PID: 1805790, Thread 1805899: 
2025-07-17T11:07:26.2571607Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2571819Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2572038Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2572241Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2572747Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2572957Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2573178Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2573272Z SIGABRT(6), PID: 1805788, Thread 1806846: 
2025-07-17T11:07:26.2573757Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2574214Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f92132ca08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2574437Z frame #2: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2574632Z frame #3: pthread_kill + 0x12c (0x7f9304bd59fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2574813Z frame #4: raise + 0x16 (0x7f9304b81476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2574982Z frame #5: abort + 0xd3 (0x7f9304b677f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2575241Z frame #6: <unknown function> + 0x972cc (0x7f9180e532cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2575547Z frame #7: <unknown function> + 0x95f47 (0x7f9180e51f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2575792Z frame #8: <unknown function> + 0x34b71 (0x7f9180df0b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2576130Z frame #9: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2576444Z frame #10: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2576450Z 
2025-07-17T11:07:26.2576454Z 
2025-07-17T11:07:26.2576553Z SIGABRT(6), PID: 1805788, Thread 1809369: 
2025-07-17T11:07:26.2577042Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2577328Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2577513Z frame #2: __poll + 0x4f (0x7f9304c57bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2577926Z frame #3: <unknown function> + 0x489f (0x7f930476389f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.2578165Z frame #4: <unknown function> + 0x1483bc (0x556988e203bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2578419Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2578659Z frame #6: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2578900Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2579129Z frame #8: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2579375Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x556988e08a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2579613Z frame #10: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2579865Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x556988e06c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2580102Z frame #12: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2580340Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2580572Z frame #14: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2580808Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2581038Z frame #16: <unknown function> + 0x147734 (0x556988e1f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2581264Z frame #17: <unknown function> + 0x21ca82 (0x556988ef4a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2581507Z frame #18: <unknown function> + 0x21c9e4 (0x556988ef49e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2581734Z frame #19: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2581955Z frame #20: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2582054Z SIGABRT(6), PID: 1805790, Thread 1805901: 
2025-07-17T11:07:26.2582540Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2582756Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2582973Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2583176Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2583692Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2583990Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2584307Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2584312Z 
2025-07-17T11:07:26.2584315Z 
2025-07-17T11:07:26.2584414Z SIGABRT(6), PID: 1805790, Thread 1805904: 
2025-07-17T11:07:26.2584957Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2585182Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2585392Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2585672Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2586176Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2586397Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2586608Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2586612Z 
2025-07-17T11:07:26.2586715Z SIGABRT(6), PID: 1805790, Thread 1805907: 
2025-07-17T11:07:26.2587190Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2587409Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2587616Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2587825Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2588325Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2588546Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2588761Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2588766Z 
2025-07-17T11:07:26.2588866Z SIGABRT(6), PID: 1805790, Thread 1805910: 
2025-07-17T11:07:26.2589339Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2589558Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2589765Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2589976Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2590474Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2590698Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2590910Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2590914Z 
2025-07-17T11:07:26.2591016Z SIGABRT(6), PID: 1805790, Thread 1805913: 
2025-07-17T11:07:26.2591493Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2591713Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2591992Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2592207Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2592767Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2593051Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2593266Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2593272Z 
2025-07-17T11:07:26.2593378Z SIGABRT(6), PID: 1805790, Thread 1805916: 
2025-07-17T11:07:26.2593848Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2594164Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2594380Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2594600Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2595102Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2595327Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2595543Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2595561Z 
2025-07-17T11:07:26.2595657Z SIGABRT(6), PID: 1805788, Thread 1809370: 
2025-07-17T11:07:26.2596142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2596356Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2596560Z frame #2: epoll_wait + 0x5e (0x7f9304c64e2e in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2596899Z frame #3: uv__io_poll + 0x6bd (0x7f925582044d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2597239Z frame #4: uv_run + 0x12a (0x7f925581436a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2597630Z frame #5: <unknown function> + 0x658a08a (0x7f9249b8808a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2597875Z frame #6: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2598092Z frame #7: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2598318Z frame #8: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2598322Z 
2025-07-17T11:07:26.2598422Z SIGABRT(6), PID: 1805790, Thread 1805919: 
2025-07-17T11:07:26.2598914Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2599128Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2599350Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2599553Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2600060Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2600271Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2600568Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2600573Z 
2025-07-17T11:07:26.2600730Z SIGABRT(6), PID: 1805790, Thread 1805922: 
2025-07-17T11:07:26.2601222Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2601493Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2601714Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2601918Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2602426Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2602704Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2602928Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2602935Z 
2025-07-17T11:07:26.2603030Z SIGABRT(6), PID: 1805790, Thread 1805925: 
2025-07-17T11:07:26.2603511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2603722Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2603941Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2604142Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2604657Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2604870Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2605097Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2605193Z SIGABRT(6), PID: 1805788, Thread 1810585: 
2025-07-17T11:07:26.2605676Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2605882Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2606096Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2606324Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f9304bd32dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2606780Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f923f012048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.2607209Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f923f014bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.2607450Z frame #6: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2607663Z frame #7: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2607882Z frame #8: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2607886Z 
2025-07-17T11:07:26.2607889Z 
2025-07-17T11:07:26.2607988Z SIGABRT(6), PID: 1805790, Thread 1805928: 
2025-07-17T11:07:26.2608459Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2608682Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2608964Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2609243Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2609812Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2610039Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2610251Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2610255Z 
2025-07-17T11:07:26.2610354Z SIGABRT(6), PID: 1805788, Thread 1810586: 
2025-07-17T11:07:26.2610886Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2611108Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2611319Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2611561Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f9304bd32dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2612042Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f923f00c741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.2612283Z frame #5: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2612492Z frame #6: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2612715Z frame #7: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2612815Z SIGABRT(6), PID: 1805790, Thread 1805931: 
2025-07-17T11:07:26.2613300Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2613513Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2613732Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2613943Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2614462Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2614678Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2614903Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2614907Z 
2025-07-17T11:07:26.2614911Z 
2025-07-17T11:07:26.2615013Z SIGABRT(6), PID: 1805788, Thread 1810588: 
2025-07-17T11:07:26.2615498Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2615707Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2615891Z frame #2: ioctl + 0x3f (0x7f9304c5994f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2616197Z frame #3: <unknown function> + 0x13eef0 (0x7f9180efaef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2616447Z frame #4: <unknown function> + 0x1378c4 (0x7f9180ef38c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2616684Z frame #5: <unknown function> + 0x95b2b (0x7f9180e51b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2616924Z frame #6: <unknown function> + 0x34b71 (0x7f9180df0b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2617245Z frame #7: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2617478Z frame #8: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2617556Z 
2025-07-17T11:07:26.2617657Z SIGABRT(6), PID: 1805790, Thread 1805933: 
2025-07-17T11:07:26.2618215Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2618434Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2618661Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2618864Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2619457Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2619672Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2619897Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2619901Z 
2025-07-17T11:07:26.2620009Z SIGABRT(6), PID: 1805790, Thread 1805935: 
2025-07-17T11:07:26.2620486Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2620708Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2620917Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2621139Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2621649Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2621895Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2622108Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2622214Z SIGABRT(6), PID: 1805788, Thread 1810602: 
2025-07-17T11:07:26.2622682Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2622903Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2623075Z frame #2: __poll + 0x4f (0x7f9304c57bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2623299Z frame #3: <unknown function> + 0xd4540 (0x7f91c906a540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2623511Z frame #4: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2623732Z frame #5: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2623738Z 
2025-07-17T11:07:26.2623742Z 
2025-07-17T11:07:26.2623837Z SIGABRT(6), PID: 1805788, Thread 1810610: 
2025-07-17T11:07:26.2624315Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2624521Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2624693Z frame #2: __poll + 0x4f (0x7f9304c57bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2624909Z frame #3: <unknown function> + 0xd65a2 (0x7f91c906c5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2625127Z frame #4: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2625410Z frame #5: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2625415Z 
2025-07-17T11:07:26.2625525Z SIGABRT(6), PID: 1805790, Thread 1805937: 
2025-07-17T11:07:26.2626058Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2626337Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2626546Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2626761Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2627259Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2627539Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2627755Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2627761Z 
2025-07-17T11:07:26.2627864Z SIGABRT(6), PID: 1805788, Thread 1810621: 
2025-07-17T11:07:26.2628333Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2628553Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2628765Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2628970Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2629184Z frame #4: <unknown function> + 0xd1e7f (0x7f91c9067e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2629396Z frame #5: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2629605Z frame #6: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2629610Z 
2025-07-17T11:07:26.2629711Z SIGABRT(6), PID: 1805790, Thread 1805938: 
2025-07-17T11:07:26.2630179Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2630397Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2630604Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2630810Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2631313Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2631525Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2631743Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2631749Z 
2025-07-17T11:07:26.2631843Z SIGABRT(6), PID: 1805788, Thread 1810625: 
2025-07-17T11:07:26.2632324Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2632532Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2632745Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2632955Z frame #3: <unknown function> + 0x9ca53 (0x7f9304bdba53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2633225Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x556988decbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2633526Z frame #5: <unknown function> + 0x1d314f (0x556988eab14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2633765Z frame #6: <unknown function> + 0x1d2f6b (0x556988eaaf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2634052Z frame #7: <unknown function> + 0x6df36 (0x556988d45f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2634367Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2634598Z frame #9: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2634849Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2635079Z frame #11: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2635388Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2635616Z frame #13: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2635864Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2636093Z frame #15: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2636347Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2636573Z frame #17: <unknown function> + 0x147734 (0x556988e1f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2636811Z frame #18: <unknown function> + 0x21ca82 (0x556988ef4a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2637036Z frame #19: <unknown function> + 0x21c9e4 (0x556988ef49e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2637269Z frame #20: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2637489Z frame #21: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2637493Z 
2025-07-17T11:07:26.2637600Z SIGABRT(6), PID: 1805790, Thread 1805940: 
2025-07-17T11:07:26.2638083Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2638315Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2638536Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2638756Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2639265Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2639502Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2639726Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2639731Z 
2025-07-17T11:07:26.2639851Z SIGABRT(6), PID: 1805790, Thread 1805943: 
2025-07-17T11:07:26.2640326Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2640562Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2640779Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2641004Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2641527Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2641826Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2642050Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2642114Z 
2025-07-17T11:07:26.2642233Z SIGABRT(6), PID: 1805788, Thread 1810641: 
2025-07-17T11:07:26.2642771Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2643006Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2643184Z frame #2: read + 0x4c (0x7f9304c5381c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2643388Z frame #3: _Py_read + 0x4c (0x556988eb2a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2643708Z frame #4: <unknown function> + 0x208c3b (0x556988ee0c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2643953Z frame #5: <unknown function> + 0x14648d (0x556988e1e48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2644210Z frame #6: PyObject_VectorcallMethod + 0x85 (0x556988e2d215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2644465Z frame #7: <unknown function> + 0x208ae7 (0x556988ee0ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2644694Z frame #8: <unknown function> + 0x208472 (0x556988ee0472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2644941Z frame #9: <unknown function> + 0x1da6ee (0x556988eb26ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2645175Z frame #10: <unknown function> + 0x1483bc (0x556988e203bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2645438Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2645674Z frame #12: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2645937Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x556988e043bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2646170Z frame #14: <unknown function> + 0x147734 (0x556988e1f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2646443Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x556988e06c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2646679Z frame #16: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2646940Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2647173Z frame #18: _PyFunction_Vectorcall + 0x6c (0x556988e140ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2647429Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x556988e047b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2647658Z frame #20: <unknown function> + 0x147734 (0x556988e1f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2647901Z frame #21: <unknown function> + 0x21ca82 (0x556988ef4a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2648131Z frame #22: <unknown function> + 0x21c9e4 (0x556988ef49e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2648371Z frame #23: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2648593Z frame #24: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2648599Z 
2025-07-17T11:07:26.2648715Z SIGABRT(6), PID: 1805790, Thread 1805945: 
2025-07-17T11:07:26.2649198Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2649430Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2649647Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2649875Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2650449Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2650750Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2651035Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2651040Z 
2025-07-17T11:07:26.2651160Z SIGABRT(6), PID: 1805790, Thread 1805947: 
2025-07-17T11:07:26.2651642Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2651937Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2652155Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2652385Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2652894Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2653133Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2653353Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2653373Z 
2025-07-17T11:07:26.2653474Z SIGABRT(6), PID: 1805790, Thread 1805949: 
2025-07-17T11:07:26.2653968Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2654190Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2654422Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2654628Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2655148Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2655370Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2655601Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2655605Z 
2025-07-17T11:07:26.2655705Z SIGABRT(6), PID: 1805790, Thread 1805951: 
2025-07-17T11:07:26.2656287Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2656508Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2656740Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2656947Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2657469Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2657689Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2657924Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2657929Z 
2025-07-17T11:07:26.2658031Z SIGABRT(6), PID: 1805790, Thread 1805954: 
2025-07-17T11:07:26.2658524Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2658830Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2659063Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2659344Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2659942Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2660164Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2660401Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2660480Z 
2025-07-17T11:07:26.2660583Z SIGABRT(6), PID: 1805790, Thread 1805956: 
2025-07-17T11:07:26.2661076Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2661293Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2661524Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2661731Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2662253Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2662477Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2662710Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2662716Z 
2025-07-17T11:07:26.2662818Z SIGABRT(6), PID: 1805790, Thread 1805959: 
2025-07-17T11:07:26.2663312Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2663531Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2663760Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2663968Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2664488Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2664722Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2664943Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2664947Z 
2025-07-17T11:07:26.2665061Z SIGABRT(6), PID: 1805788, Thread 1810702: 
2025-07-17T11:07:26.2665539Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2665770Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2665987Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2666209Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2666602Z frame #4: <unknown function> + 0x5a02023 (0x7f9249000023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2667197Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f92490041ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2667876Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9248ffb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2668350Z frame #7: <unknown function> + 0x79d655 (0x7f925a4eb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2668657Z frame #8: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2668907Z frame #9: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2669136Z frame #10: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2669140Z 
2025-07-17T11:07:26.2669326Z SIGABRT(6), PID: 1805790, Thread 1805962: 
2025-07-17T11:07:26.2669817Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2670057Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2670275Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2670504Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2671022Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2671261Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2671483Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2671491Z 
2025-07-17T11:07:26.2671611Z SIGABRT(6), PID: 1805790, Thread 1805967: 
2025-07-17T11:07:26.2672096Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2672332Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2672549Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2672778Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2673293Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2673526Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2673752Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2673756Z 
2025-07-17T11:07:26.2673871Z SIGABRT(6), PID: 1805790, Thread 1805971: 
2025-07-17T11:07:26.2674355Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2674588Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2674806Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2675031Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2675554Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2675775Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2676014Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2676017Z 
2025-07-17T11:07:26.2676119Z SIGABRT(6), PID: 1805790, Thread 1805976: 
2025-07-17T11:07:26.2676692Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2676971Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2677310Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2677526Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2678051Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2678358Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2678592Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2678596Z 
2025-07-17T11:07:26.2678703Z SIGABRT(6), PID: 1805790, Thread 1805981: 
2025-07-17T11:07:26.2679197Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2679420Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2679650Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2679858Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2680380Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2680600Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2680838Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2680842Z 
2025-07-17T11:07:26.2680943Z SIGABRT(6), PID: 1805790, Thread 1805987: 
2025-07-17T11:07:26.2681441Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2681657Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2681895Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2682102Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2682621Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2682846Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2683079Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2683085Z 
2025-07-17T11:07:26.2683184Z SIGABRT(6), PID: 1805788, Thread 1810703: 
2025-07-17T11:07:26.2683682Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2683899Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2684129Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2684340Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2684742Z frame #4: <unknown function> + 0x5a02023 (0x7f9249000023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2685393Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f92490041ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2686141Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9248ffb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2686555Z frame #7: <unknown function> + 0x79d655 (0x7f925a4eb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2686797Z frame #8: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2687040Z frame #9: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2687334Z frame #10: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2687453Z SIGABRT(6), PID: 1805790, Thread 1805992: 
2025-07-17T11:07:26.2687941Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2688173Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2688392Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2688614Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2689123Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2689362Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2689581Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2689586Z 
2025-07-17T11:07:26.2689591Z 
2025-07-17T11:07:26.2689707Z SIGABRT(6), PID: 1805790, Thread 1805997: 
2025-07-17T11:07:26.2690186Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2690418Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2690630Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2690856Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2691360Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2691590Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2691811Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2691815Z 
2025-07-17T11:07:26.2691927Z SIGABRT(6), PID: 1805794, Thread 1805794: 
2025-07-17T11:07:26.2692402Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2692637Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2692884Z frame #2: <unknown function> + 0x7744b (0x7f6bce9c344b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2693140Z frame #3: <unknown function> + 0x76c8e (0x7f6bce9c2c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2693380Z frame #4: <unknown function> + 0x6ad69 (0x7f6bce9b6d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2693628Z frame #5: <unknown function> + 0x3f54db (0x7f6c5f61b4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2693923Z frame #6: <unknown function> + 0x3e11d8 (0x7f6c5f6071d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2694227Z frame #7: <unknown function> + 0xb2f86 (0x7f6c5f2d8f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2694447Z frame #8: <unknown function> + 0x8e501 (0x7f6c5f2b4501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2694923Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f6c8a0bb7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.2695327Z frame #10: <unknown function> + 0xd307d9 (0x7f6ca860a7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2695587Z frame #11: <unknown function> + 0x139902 (0x5631ff181902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2695983Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5631ff178a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2696243Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2696501Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5631ff178a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2696756Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2697010Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2697262Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2697510Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2697762Z frame #19: <unknown function> + 0x1474b2 (0x5631ff18f4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2697982Z frame #20: PyObject_Call + 0xbc (0x5631ff18fe4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2698232Z frame #21: <unknown function> + 0x1ff69d (0x5631ff24769d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2698471Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5631ff17d023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2698742Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5631ff178cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2698978Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2699245Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5631ff1743bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2699477Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2699737Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5631ff1743bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2699972Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2700239Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5631ff178a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2700512Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5631ff17c4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2700758Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5631ff18dab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2700988Z frame #32: <unknown function> + 0x205949 (0x5631ff24d949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2701248Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5631ff17d023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2701507Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5631ff178cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2701759Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2702008Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5631ff1743bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2702254Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2702594Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5631ff1743bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2702918Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2703162Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5631ff1743bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2703488Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2703748Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5631ff1753f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2704005Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2704484Z frame #44: <unknown function> + 0x7ab3f7 (0x7f6ca80853f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2704729Z frame #45: <unknown function> + 0x13bc88 (0x5631ff183c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2704949Z frame #46: PyObject_Call + 0x20f (0x5631ff18ff9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2705220Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5631ff17998d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2705451Z frame #48: <unknown function> + 0x147608 (0x5631ff18f608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2705718Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5631ff176c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2705955Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2706215Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5631ff1743bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2706450Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2706713Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5631ff1753f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2706948Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2707210Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5631ff1743bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2707441Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2707711Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5631ff176c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2707944Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2708209Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5631ff176c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2708439Z frame #60: <unknown function> + 0x147608 (0x5631ff18f608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2708704Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5631ff176c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2708934Z frame #62: <unknown function> + 0x147608 (0x5631ff18f608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2708940Z 
2025-07-17T11:07:26.2709060Z SIGABRT(6), PID: 1805790, Thread 1806002: 
2025-07-17T11:07:26.2709555Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2709803Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2710023Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2710253Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2710779Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2711086Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2711317Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2711404Z 
2025-07-17T11:07:26.2711526Z SIGABRT(6), PID: 1805790, Thread 1806007: 
2025-07-17T11:07:26.2712070Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2712309Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2712522Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2712807Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2713315Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2713548Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2713770Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2713792Z 
2025-07-17T11:07:26.2713895Z SIGABRT(6), PID: 1805794, Thread 1806100: 
2025-07-17T11:07:26.2714391Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2714608Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2714836Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2715046Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2715570Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2715789Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2716025Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2716031Z 
2025-07-17T11:07:26.2716135Z SIGABRT(6), PID: 1805790, Thread 1806013: 
2025-07-17T11:07:26.2716629Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2716849Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2717086Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2717292Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2717810Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2718035Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2718267Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2718271Z 
2025-07-17T11:07:26.2718369Z SIGABRT(6), PID: 1805794, Thread 1806104: 
2025-07-17T11:07:26.2718863Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2719078Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2719302Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2719579Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2720095Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2720378Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2720674Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2720678Z 
2025-07-17T11:07:26.2720780Z SIGABRT(6), PID: 1805790, Thread 1806017: 
2025-07-17T11:07:26.2721276Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2721549Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2721782Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2721987Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2722503Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2722723Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2722956Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2722960Z 
2025-07-17T11:07:26.2723059Z SIGABRT(6), PID: 1805794, Thread 1806108: 
2025-07-17T11:07:26.2723550Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2723764Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2723993Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2724199Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2724719Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2724946Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2725161Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2725165Z 
2025-07-17T11:07:26.2725275Z SIGABRT(6), PID: 1805788, Thread 1810704: 
2025-07-17T11:07:26.2725744Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2725976Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2726194Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2726417Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2726804Z frame #4: <unknown function> + 0x5a02023 (0x7f9249000023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2727394Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f92490041ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2728001Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9248ffb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2728486Z frame #7: <unknown function> + 0x79d655 (0x7f925a4eb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2728726Z frame #8: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2729022Z frame #9: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2729306Z frame #10: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2729310Z 
2025-07-17T11:07:26.2729424Z SIGABRT(6), PID: 1805790, Thread 1806022: 
2025-07-17T11:07:26.2729908Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2730202Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2730418Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2730638Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2731141Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2731369Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2731590Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2731594Z 
2025-07-17T11:07:26.2731708Z SIGABRT(6), PID: 1805794, Thread 1806112: 
2025-07-17T11:07:26.2732193Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2732425Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2732638Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2732846Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2733355Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2733580Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2733791Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2733795Z 
2025-07-17T11:07:26.2733901Z SIGABRT(6), PID: 1805790, Thread 1806027: 
2025-07-17T11:07:26.2734373Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2734598Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2734808Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2735022Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2735531Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2735740Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2736026Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2736031Z 
2025-07-17T11:07:26.2736128Z SIGABRT(6), PID: 1805794, Thread 1806116: 
2025-07-17T11:07:26.2736618Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2736921Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2737142Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2737420Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2738003Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2738219Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2738443Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2738520Z 
2025-07-17T11:07:26.2738615Z SIGABRT(6), PID: 1805790, Thread 1806032: 
2025-07-17T11:07:26.2739107Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2739323Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2739541Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2739745Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2740253Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2740463Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2740689Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2740694Z 
2025-07-17T11:07:26.2740792Z SIGABRT(6), PID: 1805794, Thread 1806120: 
2025-07-17T11:07:26.2741272Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2741485Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2741714Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2741918Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2742426Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2742637Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2742863Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2742867Z 
2025-07-17T11:07:26.2742965Z SIGABRT(6), PID: 1805790, Thread 1806037: 
2025-07-17T11:07:26.2743447Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2743659Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2743879Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2744086Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2744596Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2744811Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2745044Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2745048Z 
2025-07-17T11:07:26.2745244Z SIGABRT(6), PID: 1805794, Thread 1806124: 
2025-07-17T11:07:26.2745729Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2746014Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2746282Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2746495Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2746996Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2747280Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2747496Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2747500Z 
2025-07-17T11:07:26.2747599Z SIGABRT(6), PID: 1805790, Thread 1806042: 
2025-07-17T11:07:26.2748077Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2748305Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2748515Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2748722Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2749218Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2749442Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2749656Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2749661Z 
2025-07-17T11:07:26.2749768Z SIGABRT(6), PID: 1805794, Thread 1806129: 
2025-07-17T11:07:26.2750241Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2750456Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2750662Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2750868Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2751364Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2751581Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2751796Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2751801Z 
2025-07-17T11:07:26.2751906Z SIGABRT(6), PID: 1805790, Thread 1806047: 
2025-07-17T11:07:26.2752380Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2752592Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2752801Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2753014Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2753586Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2753817Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2754096Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2754100Z 
2025-07-17T11:07:26.2754202Z SIGABRT(6), PID: 1805794, Thread 1806133: 
2025-07-17T11:07:26.2754735Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2754958Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2755163Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2755441Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2755942Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2756163Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2756377Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2756394Z 
2025-07-17T11:07:26.2756490Z SIGABRT(6), PID: 1805788, Thread 1810705: 
2025-07-17T11:07:26.2756972Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2757178Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2757399Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2757602Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2757996Z frame #4: <unknown function> + 0x5a02023 (0x7f9249000023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2758561Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f92490041ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2759168Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9248ffb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2759556Z frame #7: <unknown function> + 0x79d655 (0x7f925a4eb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2759804Z frame #8: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2760021Z frame #9: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2760250Z frame #10: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2760256Z 
2025-07-17T11:07:26.2760351Z SIGABRT(6), PID: 1805790, Thread 1806051: 
2025-07-17T11:07:26.2760837Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2761050Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2761267Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2761470Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2761982Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2762266Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2762577Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2762581Z 
2025-07-17T11:07:26.2762676Z SIGABRT(6), PID: 1805794, Thread 1806137: 
2025-07-17T11:07:26.2763224Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2763437Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2763650Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2763918Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2764428Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2764637Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2764861Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2764865Z 
2025-07-17T11:07:26.2764958Z SIGABRT(6), PID: 1805790, Thread 1806055: 
2025-07-17T11:07:26.2765443Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2765658Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2765874Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2766073Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2766591Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2766807Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2767033Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2767039Z 
2025-07-17T11:07:26.2767135Z SIGABRT(6), PID: 1805794, Thread 1806141: 
2025-07-17T11:07:26.2767616Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2767839Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2768045Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2768258Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2768757Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2768983Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2769195Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2769199Z 
2025-07-17T11:07:26.2769306Z SIGABRT(6), PID: 1805794, Thread 1806145: 
2025-07-17T11:07:26.2769777Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2769999Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2770206Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2770496Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2771000Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2771362Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2771581Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2771585Z 
2025-07-17T11:07:26.2771693Z SIGABRT(6), PID: 1805790, Thread 1806059: 
2025-07-17T11:07:26.2772167Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2772478Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2772690Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2772901Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2773402Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2773628Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2773838Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2773842Z 
2025-07-17T11:07:26.2773942Z SIGABRT(6), PID: 1805794, Thread 1806149: 
2025-07-17T11:07:26.2774410Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2774630Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2774837Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2775052Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2775552Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2775770Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2776069Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2776073Z 
2025-07-17T11:07:26.2776173Z SIGABRT(6), PID: 1805790, Thread 1806063: 
2025-07-17T11:07:26.2776648Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2776866Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2777075Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2777287Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2777787Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2778005Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2778214Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2778230Z 
2025-07-17T11:07:26.2778324Z SIGABRT(6), PID: 1805791, Thread 1805791: 
2025-07-17T11:07:26.2778894Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2779109Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2779437Z frame #2: <unknown function> + 0x7741f (0x7fa0253f341f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2779743Z frame #3: <unknown function> + 0x76c8e (0x7fa0253f2c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2779989Z frame #4: <unknown function> + 0x6ad69 (0x7fa0253e6d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2780215Z frame #5: <unknown function> + 0x3f54db (0x7fa0b604b4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2780438Z frame #6: <unknown function> + 0x3e11d8 (0x7fa0b60371d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2780735Z frame #7: <unknown function> + 0xb2f86 (0x7fa0b5d08f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2780954Z frame #8: <unknown function> + 0x8e501 (0x7fa0b5ce4501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.2781340Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fa0e0aeb7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.2781741Z frame #10: <unknown function> + 0xd307d9 (0x7fa0ff03a7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2781983Z frame #11: <unknown function> + 0x139902 (0x5561bb183902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2782252Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5561bb17aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2782494Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2782750Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5561bb17aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2782978Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2783229Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2783456Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2783707Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2783933Z frame #19: <unknown function> + 0x1474b2 (0x5561bb1914b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2784157Z frame #20: PyObject_Call + 0xbc (0x5561bb191e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2784383Z frame #21: <unknown function> + 0x1ff69d (0x5561bb24969d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2784619Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5561bb17f023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2784877Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5561bb17acc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2785120Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2785370Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5561bb1763bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2785615Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2785865Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5561bb1763bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2786105Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2786354Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5561bb17aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2786630Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5561bb17e4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2786856Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5561bb18fab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2787160Z frame #32: <unknown function> + 0x205949 (0x5561bb24f949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2787396Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5561bb17f023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2787716Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5561bb17acc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2788008Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2788264Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5561bb1763bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2788491Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2788742Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5561bb1763bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2789036Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2789289Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5561bb1763bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2789523Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2789790Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5561bb1773f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2790016Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2790436Z frame #44: <unknown function> + 0x7ab3f7 (0x7fa0feab53f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2790667Z frame #45: <unknown function> + 0x13bc88 (0x5561bb185c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2790903Z frame #46: PyObject_Call + 0x20f (0x5561bb191f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2791164Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5561bb17b98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2791399Z frame #48: <unknown function> + 0x147608 (0x5561bb191608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2791652Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5561bb178c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2791899Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2792142Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5561bb1763bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2792384Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2792629Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5561bb1773f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2792863Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2793110Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5561bb1763bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2793350Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2793595Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5561bb178c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2793833Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2794079Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5561bb178c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2794313Z frame #60: <unknown function> + 0x147608 (0x5561bb191608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2794552Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5561bb178c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2794781Z frame #62: <unknown function> + 0x147608 (0x5561bb191608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2794787Z 
2025-07-17T11:07:26.2794884Z SIGABRT(6), PID: 1805794, Thread 1806153: 
2025-07-17T11:07:26.2795465Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2795696Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2795981Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2796262Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2796785Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2797007Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2797298Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2797302Z 
2025-07-17T11:07:26.2797400Z SIGABRT(6), PID: 1805790, Thread 1806067: 
2025-07-17T11:07:26.2797890Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2798108Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2798332Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2798538Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2799055Z frame #4: <unknown function> + 0x3475ab (0x7fa04bf795ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2799266Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2799497Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2799501Z 
2025-07-17T11:07:26.2799593Z SIGABRT(6), PID: 1805791, Thread 1805966: 
2025-07-17T11:07:26.2800077Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2800288Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2800503Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2800704Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2801216Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2801431Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2801649Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2801746Z SIGABRT(6), PID: 1805790, Thread 1806848: 
2025-07-17T11:07:26.2802229Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2802692Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f9f5d7da08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2802912Z frame #2: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2803109Z frame #3: pthread_kill + 0x12c (0x7fa04f0e39fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2803290Z frame #4: raise + 0x16 (0x7fa04f08f476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2803457Z frame #5: abort + 0xd3 (0x7fa04f0757f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2803711Z frame #6: <unknown function> + 0x972cc (0x7f9ecb3632cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2804025Z frame #7: <unknown function> + 0x95f47 (0x7f9ecb361f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2804327Z frame #8: <unknown function> + 0x34b71 (0x7f9ecb300b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2804549Z frame #9: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2804863Z frame #10: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2804966Z SIGABRT(6), PID: 1805794, Thread 1806157: 
2025-07-17T11:07:26.2805460Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2805741Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2805961Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2806172Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2806695Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2806912Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2807135Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2807228Z SIGABRT(6), PID: 1805788, Thread 1810706: 
2025-07-17T11:07:26.2807719Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2807935Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2808151Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2808355Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2808752Z frame #4: <unknown function> + 0x5a02023 (0x7f9249000023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2809330Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f92490041ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2809941Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9248ffb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2810342Z frame #7: <unknown function> + 0x79d655 (0x7f925a4eb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2810577Z frame #8: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2810803Z frame #9: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2811022Z frame #10: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2811027Z 
2025-07-17T11:07:26.2811030Z 
2025-07-17T11:07:26.2811033Z 
2025-07-17T11:07:26.2811038Z 
2025-07-17T11:07:26.2811145Z SIGABRT(6), PID: 1805794, Thread 1806161: 
2025-07-17T11:07:26.2811626Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2811850Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2821902Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2822202Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2822904Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2823228Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2823530Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2823536Z 
2025-07-17T11:07:26.2823639Z SIGABRT(6), PID: 1805790, Thread 1809170: 
2025-07-17T11:07:26.2824137Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2824444Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2824618Z frame #2: __poll + 0x4f (0x7fa04f165bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2825038Z frame #3: <unknown function> + 0x489f (0x7fa04ec7189f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.2825279Z frame #4: <unknown function> + 0x1483bc (0x56525bf1f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2825531Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2825757Z frame #6: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2825998Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2826230Z frame #8: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2826485Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x56525bf07a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2826712Z frame #10: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2826967Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x56525bf05c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2827192Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2827429Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2827653Z frame #14: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2827887Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2828107Z frame #16: <unknown function> + 0x147734 (0x56525bf1e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2828330Z frame #17: <unknown function> + 0x21ca82 (0x56525bff3a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2828545Z frame #18: <unknown function> + 0x21c9e4 (0x56525bff39e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2828767Z frame #19: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2828983Z frame #20: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2828987Z 
2025-07-17T11:07:26.2829087Z SIGABRT(6), PID: 1805791, Thread 1805972: 
2025-07-17T11:07:26.2829571Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2829785Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2829992Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2830202Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2830791Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2831007Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2831280Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2831285Z 
2025-07-17T11:07:26.2831452Z SIGABRT(6), PID: 1805794, Thread 1806166: 
2025-07-17T11:07:26.2831935Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2832141Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2832349Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2832616Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2833122Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2833330Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2833538Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2833544Z 
2025-07-17T11:07:26.2833633Z SIGABRT(6), PID: 1805791, Thread 1805978: 
2025-07-17T11:07:26.2834105Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2834307Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2834512Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2834707Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2835206Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2835411Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2835626Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2835631Z 
2025-07-17T11:07:26.2835719Z SIGABRT(6), PID: 1805794, Thread 1806170: 
2025-07-17T11:07:26.2836185Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2836391Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2836595Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2836789Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2837287Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2837495Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2837703Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2837708Z 
2025-07-17T11:07:26.2837793Z SIGABRT(6), PID: 1805791, Thread 1805984: 
2025-07-17T11:07:26.2838259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2838467Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2838742Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2838944Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2839504Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2839768Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2839982Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2839986Z 
2025-07-17T11:07:26.2840072Z SIGABRT(6), PID: 1805794, Thread 1806174: 
2025-07-17T11:07:26.2840544Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2840815Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2841025Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2841230Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2841734Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2841948Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2842164Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2842168Z 
2025-07-17T11:07:26.2842260Z SIGABRT(6), PID: 1805790, Thread 1809567: 
2025-07-17T11:07:26.2842750Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2842971Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2843174Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2843404Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa04f0e12dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2843849Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f9f89522048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.2844277Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f9f89524bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.2844513Z frame #6: <unknown function> + 0xdc253 (0x7f9f5ba46253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2844730Z frame #7: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2844939Z frame #8: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2844943Z 
2025-07-17T11:07:26.2845037Z SIGABRT(6), PID: 1805791, Thread 1805989: 
2025-07-17T11:07:26.2845511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2845723Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2845922Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2846123Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2846626Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2846836Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2847112Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2847196Z 
2025-07-17T11:07:26.2847294Z SIGABRT(6), PID: 1805794, Thread 1806179: 
2025-07-17T11:07:26.2847825Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2848038Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2848239Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2848441Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2849004Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2849211Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2849418Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2849424Z 
2025-07-17T11:07:26.2849519Z SIGABRT(6), PID: 1805791, Thread 1805994: 
2025-07-17T11:07:26.2849988Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2850193Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2850392Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2850590Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2851085Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2851294Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2851503Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2851508Z 
2025-07-17T11:07:26.2851599Z SIGABRT(6), PID: 1805794, Thread 1806183: 
2025-07-17T11:07:26.2852065Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2852273Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2852473Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2852671Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2853163Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2853370Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2853577Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2853669Z SIGABRT(6), PID: 1805790, Thread 1809568: 
2025-07-17T11:07:26.2854133Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2854337Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2854546Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2854766Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa04f0e12dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2855329Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f9f8951c741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.2855624Z frame #5: <unknown function> + 0xdc253 (0x7f9f5ba46253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2855900Z frame #6: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2856186Z frame #7: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2856190Z 
2025-07-17T11:07:26.2856282Z SIGABRT(6), PID: 1805788, Thread 1810707: 
2025-07-17T11:07:26.2856742Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2857031Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2857235Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2857435Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2857815Z frame #4: <unknown function> + 0x5a02023 (0x7f9249000023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2858384Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f92490041ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2858978Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9248ffb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2859365Z frame #7: <unknown function> + 0x79d655 (0x7f925a4eb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2859597Z frame #8: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2859814Z frame #9: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2860030Z frame #10: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2860035Z 
2025-07-17T11:07:26.2860038Z 
2025-07-17T11:07:26.2860134Z SIGABRT(6), PID: 1805791, Thread 1805999: 
2025-07-17T11:07:26.2860607Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2860818Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2861020Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2861225Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2861724Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2861934Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2862146Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2862150Z 
2025-07-17T11:07:26.2862241Z SIGABRT(6), PID: 1805790, Thread 1809569: 
2025-07-17T11:07:26.2862724Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2862933Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2863100Z frame #2: ioctl + 0x3f (0x7fa04f16794f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2863430Z frame #3: <unknown function> + 0x13eef0 (0x7f9ecb40aef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2863668Z frame #4: <unknown function> + 0x1378c4 (0x7f9ecb4038c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2863979Z frame #5: <unknown function> + 0x95b2b (0x7f9ecb361b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2864278Z frame #6: <unknown function> + 0x34b71 (0x7f9ecb300b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.2864494Z frame #7: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2864704Z frame #8: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2864711Z 
2025-07-17T11:07:26.2864800Z SIGABRT(6), PID: 1805794, Thread 1806187: 
2025-07-17T11:07:26.2865338Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2865546Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2865749Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2865951Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2866458Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2866665Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2866874Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2866880Z 
2025-07-17T11:07:26.2866970Z SIGABRT(6), PID: 1805791, Thread 1806004: 
2025-07-17T11:07:26.2867436Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2867642Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2867848Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2868041Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2868539Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2868742Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2868950Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2868955Z 
2025-07-17T11:07:26.2869042Z SIGABRT(6), PID: 1805790, Thread 1810604: 
2025-07-17T11:07:26.2869513Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2869719Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2869890Z frame #2: __poll + 0x4f (0x7fa04f165bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2870102Z frame #3: <unknown function> + 0xd4540 (0x7f9f1357a540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2870308Z frame #4: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2870513Z frame #5: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2870516Z 
2025-07-17T11:07:26.2870608Z SIGABRT(6), PID: 1805794, Thread 1806191: 
2025-07-17T11:07:26.2871073Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2871351Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2871558Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2871819Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2872371Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2872586Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2872793Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2872856Z 
2025-07-17T11:07:26.2872950Z SIGABRT(6), PID: 1805791, Thread 1806010: 
2025-07-17T11:07:26.2873414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2873621Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2873827Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2874025Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2874523Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2874729Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2874934Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2874940Z 
2025-07-17T11:07:26.2875031Z SIGABRT(6), PID: 1805790, Thread 1810611: 
2025-07-17T11:07:26.2875495Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2875701Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2875865Z frame #2: __poll + 0x4f (0x7fa04f165bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2876078Z frame #3: <unknown function> + 0xd65a2 (0x7f9f1357c5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2876282Z frame #4: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2876493Z frame #5: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2876497Z 
2025-07-17T11:07:26.2876583Z SIGABRT(6), PID: 1805794, Thread 1806197: 
2025-07-17T11:07:26.2877047Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2877257Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2877457Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2877653Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2878145Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2878352Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2878558Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2878563Z 
2025-07-17T11:07:26.2878654Z SIGABRT(6), PID: 1805790, Thread 1810614: 
2025-07-17T11:07:26.2879210Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2879423Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2879686Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2879884Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2880149Z frame #4: <unknown function> + 0xd1e7f (0x7f9f13577e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.2880359Z frame #5: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2880565Z frame #6: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2880632Z 
2025-07-17T11:07:26.2880728Z SIGABRT(6), PID: 1805791, Thread 1806015: 
2025-07-17T11:07:26.2881196Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2881403Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2881606Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2881806Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2882304Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2882512Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2882714Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2882720Z 
2025-07-17T11:07:26.2882811Z SIGABRT(6), PID: 1805794, Thread 1806202: 
2025-07-17T11:07:26.2883277Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2883485Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2883685Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2883880Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2884374Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2884580Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2884786Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2884790Z 
2025-07-17T11:07:26.2884878Z SIGABRT(6), PID: 1805790, Thread 1810624: 
2025-07-17T11:07:26.2885339Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2885550Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2885752Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2885957Z frame #3: <unknown function> + 0x9ca53 (0x7fa04f0e9a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2886209Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x56525beebbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2886437Z frame #5: <unknown function> + 0x1d314f (0x56525bfaa14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2886660Z frame #6: <unknown function> + 0x1d2f6b (0x56525bfa9f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2886883Z frame #7: <unknown function> + 0x6df36 (0x56525be44f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2887199Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2887487Z frame #9: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2887786Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2888016Z frame #11: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2888246Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2888469Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2888763Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2888986Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2889214Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2889434Z frame #17: <unknown function> + 0x147734 (0x56525bf1e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2889650Z frame #18: <unknown function> + 0x21ca82 (0x56525bff3a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2889870Z frame #19: <unknown function> + 0x21c9e4 (0x56525bff39e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2890080Z frame #20: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2890292Z frame #21: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2890382Z SIGABRT(6), PID: 1805791, Thread 1806020: 
2025-07-17T11:07:26.2890849Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2891056Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2891263Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2891460Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2891961Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2892166Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2892374Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2892381Z 
2025-07-17T11:07:26.2892384Z 
2025-07-17T11:07:26.2892474Z SIGABRT(6), PID: 1805794, Thread 1806207: 
2025-07-17T11:07:26.2892940Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2893147Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2893346Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2893543Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2894036Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2894245Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2894452Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2894456Z 
2025-07-17T11:07:26.2894544Z SIGABRT(6), PID: 1805791, Thread 1806025: 
2025-07-17T11:07:26.2895079Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2895347Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2895608Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2895812Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2896369Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2896658Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2896864Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2896869Z 
2025-07-17T11:07:26.2896960Z SIGABRT(6), PID: 1805790, Thread 1810633: 
2025-07-17T11:07:26.2897423Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2897637Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2897801Z frame #2: read + 0x4c (0x7fa04f16181c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2897979Z frame #3: _Py_read + 0x4c (0x56525bfb1a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2898205Z frame #4: <unknown function> + 0x208c3b (0x56525bfdfc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2898427Z frame #5: <unknown function> + 0x14648d (0x56525bf1d48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2898673Z frame #6: PyObject_VectorcallMethod + 0x85 (0x56525bf2c215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2898897Z frame #7: <unknown function> + 0x208ae7 (0x56525bfdfae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2899110Z frame #8: <unknown function> + 0x208472 (0x56525bfdf472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2899331Z frame #9: <unknown function> + 0x1da6ee (0x56525bfb16ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2899558Z frame #10: <unknown function> + 0x1483bc (0x56525bf1f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2899799Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2900020Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2900254Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x56525bf033bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2900474Z frame #14: <unknown function> + 0x147734 (0x56525bf1e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2900720Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x56525bf05c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2900941Z frame #16: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2901175Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2901396Z frame #18: _PyFunction_Vectorcall + 0x6c (0x56525bf130ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2901628Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x56525bf037b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2901842Z frame #20: <unknown function> + 0x147734 (0x56525bf1e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2902061Z frame #21: <unknown function> + 0x21ca82 (0x56525bff3a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2902279Z frame #22: <unknown function> + 0x21c9e4 (0x56525bff39e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.2902491Z frame #23: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2902785Z frame #24: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2902860Z 
2025-07-17T11:07:26.2902958Z SIGABRT(6), PID: 1805794, Thread 1806212: 
2025-07-17T11:07:26.2903510Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2903727Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2903929Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2904129Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2904711Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2904918Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2905125Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2905129Z 
2025-07-17T11:07:26.2905220Z SIGABRT(6), PID: 1805788, Thread 1810708: 
2025-07-17T11:07:26.2905682Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2905887Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2906089Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2906291Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2906663Z frame #4: <unknown function> + 0x5a02023 (0x7f9249000023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2907227Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f92490041ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2907823Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9248ffb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2908205Z frame #7: <unknown function> + 0x79d655 (0x7f925a4eb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2908432Z frame #8: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2908640Z frame #9: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2908854Z frame #10: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2908860Z 
2025-07-17T11:07:26.2908948Z SIGABRT(6), PID: 1805791, Thread 1806029: 
2025-07-17T11:07:26.2909422Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2909628Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2909829Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2910024Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2910522Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2910729Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2911017Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2911022Z 
2025-07-17T11:07:26.2911170Z SIGABRT(6), PID: 1805794, Thread 1806216: 
2025-07-17T11:07:26.2911640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2911904Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2912124Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2912318Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2912880Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2913084Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2913291Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2913297Z 
2025-07-17T11:07:26.2913387Z SIGABRT(6), PID: 1805794, Thread 1806219: 
2025-07-17T11:07:26.2913854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2914060Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2914261Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2914455Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2914956Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2915162Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2915375Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2915381Z 
2025-07-17T11:07:26.2915466Z SIGABRT(6), PID: 1805791, Thread 1806034: 
2025-07-17T11:07:26.2915939Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2916142Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2916347Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2916544Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2917041Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2917249Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2917458Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2917462Z 
2025-07-17T11:07:26.2917551Z SIGABRT(6), PID: 1805791, Thread 1806038: 
2025-07-17T11:07:26.2918019Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2918226Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2918430Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2918632Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2919196Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2919471Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2919737Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2919831Z SIGABRT(6), PID: 1805794, Thread 1806222: 
2025-07-17T11:07:26.2920296Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2920502Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2920768Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2920964Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2921454Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2921664Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2921871Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2921875Z 
2025-07-17T11:07:26.2921879Z 
2025-07-17T11:07:26.2921970Z SIGABRT(6), PID: 1805791, Thread 1806041: 
2025-07-17T11:07:26.2922430Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2922637Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2922836Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2923043Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2923538Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2923751Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2923956Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2923960Z 
2025-07-17T11:07:26.2924053Z SIGABRT(6), PID: 1805794, Thread 1806224: 
2025-07-17T11:07:26.2924511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2924720Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2924922Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2925123Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2925617Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2925826Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2926033Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2926036Z 
2025-07-17T11:07:26.2926127Z SIGABRT(6), PID: 1805791, Thread 1806045: 
2025-07-17T11:07:26.2926593Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2926800Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2927079Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2927336Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2927885Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2928096Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2928304Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2928313Z 
2025-07-17T11:07:26.2928482Z SIGABRT(6), PID: 1805794, Thread 1806226: 
2025-07-17T11:07:26.2928954Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2929159Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2929368Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2929564Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2930058Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2930264Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2930474Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2930480Z 
2025-07-17T11:07:26.2930568Z SIGABRT(6), PID: 1805790, Thread 1810654: 
2025-07-17T11:07:26.2931035Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2931245Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2931457Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2931657Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2932038Z frame #4: <unknown function> + 0x5a02023 (0x7f9f93510023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2932599Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9f935141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2933202Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9f9350b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2933581Z frame #7: <unknown function> + 0x79d655 (0x7f9fa49fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2933814Z frame #8: <unknown function> + 0xdc253 (0x7f9f5ba46253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2934029Z frame #9: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2934252Z frame #10: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2934257Z 
2025-07-17T11:07:26.2934345Z SIGABRT(6), PID: 1805791, Thread 1806049: 
2025-07-17T11:07:26.2934822Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2935031Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2935315Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2935517Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2936258Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2936478Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2939007Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2939019Z 
2025-07-17T11:07:26.2939146Z SIGABRT(6), PID: 1805794, Thread 1806229: 
2025-07-17T11:07:26.2939775Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2940009Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2940227Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2940523Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2941090Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2941357Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2941671Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2941677Z 
2025-07-17T11:07:26.2941827Z SIGABRT(6), PID: 1805788, Thread 1810709: 
2025-07-17T11:07:26.2942340Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f92132c9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2942553Z frame #1: <unknown function> + 0x42520 (0x7f9304b81520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2942768Z frame #2: <unknown function> + 0x91117 (0x7f9304bd0117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2942975Z frame #3: pthread_cond_wait + 0x211 (0x7f9304bd2a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2943361Z frame #4: <unknown function> + 0x5a02023 (0x7f9249000023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2943966Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f92490041ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2944578Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9248ffb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2944975Z frame #7: <unknown function> + 0x79d655 (0x7f925a4eb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2945214Z frame #8: <unknown function> + 0xdc253 (0x7f9211536253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2945430Z frame #9: <unknown function> + 0x94ac3 (0x7f9304bd3ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2945648Z frame #10: <unknown function> + 0x126850 (0x7f9304c65850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2945653Z 
2025-07-17T11:07:26.2945753Z SIGABRT(6), PID: 1805791, Thread 1806053: 
2025-07-17T11:07:26.2946237Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2946454Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2946668Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2946874Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2947500Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2947777Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2948012Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2948016Z 
2025-07-17T11:07:26.2948254Z SIGABRT(6), PID: 1805794, Thread 1806231: 
2025-07-17T11:07:26.2948742Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2949040Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2949255Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2949471Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2949980Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2950196Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2950436Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2950440Z 
2025-07-17T11:07:26.2950548Z SIGABRT(6), PID: 1805794, Thread 1806233: 
2025-07-17T11:07:26.2951029Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2951248Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2951451Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2951661Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2952161Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2952383Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2952604Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2952613Z 
2025-07-17T11:07:26.2952704Z SIGABRT(6), PID: 1805791, Thread 1806057: 
2025-07-17T11:07:26.2953201Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2953406Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2953617Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2953821Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2954327Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2954535Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2954749Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2954755Z 
2025-07-17T11:07:26.2954845Z SIGABRT(6), PID: 1805794, Thread 1806235: 
2025-07-17T11:07:26.2955321Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2955619Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2955830Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2956096Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2956670Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2956882Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2957161Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2957165Z 
2025-07-17T11:07:26.2957256Z SIGABRT(6), PID: 1805791, Thread 1806061: 
2025-07-17T11:07:26.2957736Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2957947Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2958164Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2958362Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2958860Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2959068Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2959282Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2959286Z 
2025-07-17T11:07:26.2959377Z SIGABRT(6), PID: 1805794, Thread 1806237: 
2025-07-17T11:07:26.2959848Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2960060Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2960269Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2960466Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2960964Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2961176Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2961389Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2961393Z 
2025-07-17T11:07:26.2961486Z SIGABRT(6), PID: 1805791, Thread 1806064: 
2025-07-17T11:07:26.2961960Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2962177Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2962388Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2962584Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2963088Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2963297Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2963510Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2963588Z 
2025-07-17T11:07:26.2963687Z SIGABRT(6), PID: 1805794, Thread 1806239: 
2025-07-17T11:07:26.2964221Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2964436Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2964639Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2964903Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2965466Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2965676Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2965886Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2965891Z 
2025-07-17T11:07:26.2965987Z SIGABRT(6), PID: 1805791, Thread 1806068: 
2025-07-17T11:07:26.2966460Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2966668Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2966877Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2967083Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2967579Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2967790Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2968002Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2968006Z 
2025-07-17T11:07:26.2968100Z SIGABRT(6), PID: 1805794, Thread 1806241: 
2025-07-17T11:07:26.2968567Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2968778Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2968983Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2969188Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2969682Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2969896Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2970105Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2970110Z 
2025-07-17T11:07:26.2970207Z SIGABRT(6), PID: 1805790, Thread 1810655: 
2025-07-17T11:07:26.2970673Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2970888Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2971096Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2971298Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2971688Z frame #4: <unknown function> + 0x5a02023 (0x7f9f93510023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2972371Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9f935141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2973026Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9f9350b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.2973481Z frame #7: <unknown function> + 0x79d655 (0x7f9fa49fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.2973783Z frame #8: <unknown function> + 0xdc253 (0x7f9f5ba46253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.2973995Z frame #9: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2974216Z frame #10: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2974229Z 
2025-07-17T11:07:26.2974321Z SIGABRT(6), PID: 1805791, Thread 1806071: 
2025-07-17T11:07:26.2974803Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2975008Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2975219Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2975421Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2976006Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2976217Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2976433Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2976438Z 
2025-07-17T11:07:26.2976530Z SIGABRT(6), PID: 1805794, Thread 1806243: 
2025-07-17T11:07:26.2977011Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2977219Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2977432Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2977634Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2978137Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2978343Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2978558Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2978563Z 
2025-07-17T11:07:26.2978652Z SIGABRT(6), PID: 1805794, Thread 1806245: 
2025-07-17T11:07:26.2979124Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2979332Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2979539Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2979740Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2980240Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2980543Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2980760Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2980838Z 
2025-07-17T11:07:26.2980933Z SIGABRT(6), PID: 1805791, Thread 1806074: 
2025-07-17T11:07:26.2981480Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2981693Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2981978Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2982186Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2982686Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2982895Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2983109Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2983113Z 
2025-07-17T11:07:26.2983203Z SIGABRT(6), PID: 1805791, Thread 1806077: 
2025-07-17T11:07:26.2983676Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2983882Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2984092Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2984287Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2984785Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2984999Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2985206Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2985209Z 
2025-07-17T11:07:26.2985303Z SIGABRT(6), PID: 1805794, Thread 1806247: 
2025-07-17T11:07:26.2985773Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2985984Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2986188Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2986388Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2986885Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2987100Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2987310Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2987314Z 
2025-07-17T11:07:26.2987410Z SIGABRT(6), PID: 1805791, Thread 1806081: 
2025-07-17T11:07:26.2987875Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2988086Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2988287Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2988569Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2989125Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2989345Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2989610Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2989615Z 
2025-07-17T11:07:26.2989711Z SIGABRT(6), PID: 1805794, Thread 1806249: 
2025-07-17T11:07:26.2990252Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2990462Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2990664Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2990869Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2991373Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2991591Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2991801Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2991806Z 
2025-07-17T11:07:26.2991904Z SIGABRT(6), PID: 1805794, Thread 1806251: 
2025-07-17T11:07:26.2992376Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2992585Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2992793Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2992994Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2993492Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2993709Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2993920Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2994024Z SIGABRT(6), PID: 1805791, Thread 1806084: 
2025-07-17T11:07:26.2994498Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2994705Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2994911Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2995118Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2995612Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2995828Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2996040Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2996049Z 
2025-07-17T11:07:26.2996053Z 
2025-07-17T11:07:26.2996143Z SIGABRT(6), PID: 1805791, Thread 1806087: 
2025-07-17T11:07:26.2996615Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2996900Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2997173Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2997374Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2997939Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.2998232Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2998441Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2998446Z 
2025-07-17T11:07:26.2998537Z SIGABRT(6), PID: 1805794, Thread 1806254: 
2025-07-17T11:07:26.2999009Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.2999222Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2999431Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.2999630Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3000132Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3000345Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3000555Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3000559Z 
2025-07-17T11:07:26.3000650Z SIGABRT(6), PID: 1805790, Thread 1810656: 
2025-07-17T11:07:26.3001124Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3001330Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3001539Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3001740Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3002115Z frame #4: <unknown function> + 0x5a02023 (0x7f9f93510023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3002688Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9f935141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3003285Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9f9350b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3003671Z frame #7: <unknown function> + 0x79d655 (0x7f9fa49fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3003911Z frame #8: <unknown function> + 0xdc253 (0x7f9f5ba46253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3004122Z frame #9: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3004345Z frame #10: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3004349Z 
2025-07-17T11:07:26.3004441Z SIGABRT(6), PID: 1805791, Thread 1806090: 
2025-07-17T11:07:26.3004916Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3005204Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3005408Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3005688Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3006257Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3006478Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3006752Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3006756Z 
2025-07-17T11:07:26.3006855Z SIGABRT(6), PID: 1805794, Thread 1806256: 
2025-07-17T11:07:26.3007323Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3007541Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3007750Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3007950Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3008448Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3008665Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3008881Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3008885Z 
2025-07-17T11:07:26.3008985Z SIGABRT(6), PID: 1805791, Thread 1806093: 
2025-07-17T11:07:26.3009461Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3009675Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3009881Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3010089Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3010589Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3010814Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3011022Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3011030Z 
2025-07-17T11:07:26.3011123Z SIGABRT(6), PID: 1805794, Thread 1806257: 
2025-07-17T11:07:26.3011601Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3011809Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3012024Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3012226Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3012727Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3012937Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3013150Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3013230Z 
2025-07-17T11:07:26.3013326Z SIGABRT(6), PID: 1805791, Thread 1806096: 
2025-07-17T11:07:26.3013887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3014100Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3014371Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3014575Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3015135Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3015345Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3015566Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3015570Z 
2025-07-17T11:07:26.3015663Z SIGABRT(6), PID: 1805794, Thread 1806259: 
2025-07-17T11:07:26.3016210Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3016419Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3016632Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3016836Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3017336Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3017545Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3017757Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3017762Z 
2025-07-17T11:07:26.3017857Z SIGABRT(6), PID: 1805794, Thread 1806262: 
2025-07-17T11:07:26.3018333Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3018544Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3018757Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3018960Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3019469Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3019680Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3019894Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3019899Z 
2025-07-17T11:07:26.3019994Z SIGABRT(6), PID: 1805791, Thread 1806099: 
2025-07-17T11:07:26.3020474Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3020682Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3020894Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3021090Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3021595Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3021904Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3022192Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3022196Z 
2025-07-17T11:07:26.3022297Z SIGABRT(6), PID: 1805794, Thread 1806264: 
2025-07-17T11:07:26.3022838Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3023121Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3023328Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3023529Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3024031Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3024251Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3024461Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3024559Z SIGABRT(6), PID: 1805791, Thread 1806103: 
2025-07-17T11:07:26.3025036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3025251Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3025454Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3025663Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3026160Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3026376Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3026587Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3026591Z 
2025-07-17T11:07:26.3026594Z 
2025-07-17T11:07:26.3026692Z SIGABRT(6), PID: 1805794, Thread 1806266: 
2025-07-17T11:07:26.3027161Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3027381Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3027583Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3027789Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3028284Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3028499Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3028711Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3028715Z 
2025-07-17T11:07:26.3028811Z SIGABRT(6), PID: 1805791, Thread 1806106: 
2025-07-17T11:07:26.3029284Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3029492Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3029698Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3029981Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3030541Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3030759Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3031053Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3031114Z 
2025-07-17T11:07:26.3031215Z SIGABRT(6), PID: 1805790, Thread 1810657: 
2025-07-17T11:07:26.3031686Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3031898Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3032109Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3032311Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3032687Z frame #4: <unknown function> + 0x5a02023 (0x7f9f93510023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3033258Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9f935141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3033858Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9f9350b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3034243Z frame #7: <unknown function> + 0x79d655 (0x7f9fa49fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3034488Z frame #8: <unknown function> + 0xdc253 (0x7f9f5ba46253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3034697Z frame #9: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3034921Z frame #10: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3034925Z 
2025-07-17T11:07:26.3035017Z SIGABRT(6), PID: 1805794, Thread 1806268: 
2025-07-17T11:07:26.3035499Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3035708Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3035917Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3036116Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3036628Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3036837Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3037048Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3037054Z 
2025-07-17T11:07:26.3037144Z SIGABRT(6), PID: 1805791, Thread 1806109: 
2025-07-17T11:07:26.3037622Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3037829Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3038038Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3038314Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3038879Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3039094Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3039373Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3039378Z 
2025-07-17T11:07:26.3039531Z SIGABRT(6), PID: 1805794, Thread 1806270: 
2025-07-17T11:07:26.3040009Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3040218Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3040432Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3040628Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3041131Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3041341Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3041556Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3041562Z 
2025-07-17T11:07:26.3041654Z SIGABRT(6), PID: 1805791, Thread 1806113: 
2025-07-17T11:07:26.3042126Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3042349Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3042557Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3042763Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3043257Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3043472Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3043682Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3043686Z 
2025-07-17T11:07:26.3043780Z SIGABRT(6), PID: 1805794, Thread 1806272: 
2025-07-17T11:07:26.3044247Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3044460Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3044665Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3044869Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3045363Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3045577Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3045786Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3045790Z 
2025-07-17T11:07:26.3045886Z SIGABRT(6), PID: 1805791, Thread 1806117: 
2025-07-17T11:07:26.3046354Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3046646Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3046912Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3047121Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3047709Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3047986Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3048196Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3048200Z 
2025-07-17T11:07:26.3048294Z SIGABRT(6), PID: 1805794, Thread 1806274: 
2025-07-17T11:07:26.3048764Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3048980Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3049186Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3049389Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3049885Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3050101Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3050309Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3050313Z 
2025-07-17T11:07:26.3050408Z SIGABRT(6), PID: 1805791, Thread 1806121: 
2025-07-17T11:07:26.3050876Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3051087Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3051293Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3051492Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3051988Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3052203Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3052411Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3052424Z 
2025-07-17T11:07:26.3052517Z SIGABRT(6), PID: 1805791, Thread 1806125: 
2025-07-17T11:07:26.3052991Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3053198Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3053412Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3053616Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3054122Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3054330Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3054623Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3054627Z 
2025-07-17T11:07:26.3054717Z SIGABRT(6), PID: 1805794, Thread 1806276: 
2025-07-17T11:07:26.3055255Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3055466Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3055735Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3056051Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3056558Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3056766Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3056980Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3056984Z 
2025-07-17T11:07:26.3057075Z SIGABRT(6), PID: 1805794, Thread 1806277: 
2025-07-17T11:07:26.3057552Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3057763Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3057973Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3058171Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3058672Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3058882Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3059094Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3059100Z 
2025-07-17T11:07:26.3059188Z SIGABRT(6), PID: 1805791, Thread 1806128: 
2025-07-17T11:07:26.3059663Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3059872Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3060079Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3060277Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3060778Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3060988Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3061204Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3061208Z 
2025-07-17T11:07:26.3061298Z SIGABRT(6), PID: 1805790, Thread 1810658: 
2025-07-17T11:07:26.3061776Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3061983Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3062191Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3062388Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3062771Z frame #4: <unknown function> + 0x5a02023 (0x7f9f93510023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3063529Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9f935141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3064207Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9f9350b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3064672Z frame #7: <unknown function> + 0x79d655 (0x7f9fa49fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3064908Z frame #8: <unknown function> + 0xdc253 (0x7f9f5ba46253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3065120Z frame #9: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3065339Z frame #10: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3065343Z 
2025-07-17T11:07:26.3065442Z SIGABRT(6), PID: 1805794, Thread 1806279: 
2025-07-17T11:07:26.3065915Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3066129Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3066334Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3066537Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3067035Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3067250Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3067461Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3067465Z 
2025-07-17T11:07:26.3067562Z SIGABRT(6), PID: 1805791, Thread 1806132: 
2025-07-17T11:07:26.3068036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3068246Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3068474Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3068681Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3069179Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3069393Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3069604Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3069609Z 
2025-07-17T11:07:26.3069705Z SIGABRT(6), PID: 1805794, Thread 1806281: 
2025-07-17T11:07:26.3070174Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3070389Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3070598Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3070801Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3071293Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3071591Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3071870Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3071875Z 
2025-07-17T11:07:26.3071976Z SIGABRT(6), PID: 1805791, Thread 1806136: 
2025-07-17T11:07:26.3072508Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3072782Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3072987Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3073193Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3073688Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3073906Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3074114Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3074125Z 
2025-07-17T11:07:26.3074217Z SIGABRT(6), PID: 1805794, Thread 1806283: 
2025-07-17T11:07:26.3074694Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3074901Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3075111Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3075310Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3075813Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3076021Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3076236Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3076242Z 
2025-07-17T11:07:26.3076333Z SIGABRT(6), PID: 1805791, Thread 1806140: 
2025-07-17T11:07:26.3076816Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3077022Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3077232Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3077435Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3077941Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3078148Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3078363Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3078367Z 
2025-07-17T11:07:26.3078459Z SIGABRT(6), PID: 1805794, Thread 1806284: 
2025-07-17T11:07:26.3078935Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3079141Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3079426Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3079625Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3080186Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3080454Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3080673Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3080735Z 
2025-07-17T11:07:26.3080828Z SIGABRT(6), PID: 1805791, Thread 1806144: 
2025-07-17T11:07:26.3081305Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3081513Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3081723Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3081924Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3082429Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3082639Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3082853Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3082857Z 
2025-07-17T11:07:26.3082944Z SIGABRT(6), PID: 1805794, Thread 1806286: 
2025-07-17T11:07:26.3083418Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3083628Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3083835Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3084037Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3084537Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3084743Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3084954Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3084959Z 
2025-07-17T11:07:26.3085055Z SIGABRT(6), PID: 1805791, Thread 1806148: 
2025-07-17T11:07:26.3085527Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3085744Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3085949Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3086149Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3086643Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3086863Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3087074Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3087078Z 
2025-07-17T11:07:26.3087173Z SIGABRT(6), PID: 1805794, Thread 1806287: 
2025-07-17T11:07:26.3087722Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3087994Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3088203Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3088401Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3088951Z frame #4: <unknown function> + 0x3475ab (0x7f6d4f5f95ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3089228Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3089440Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3089444Z 
2025-07-17T11:07:26.3089542Z SIGABRT(6), PID: 1805790, Thread 1810659: 
2025-07-17T11:07:26.3090012Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3090226Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3090431Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3090636Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3091010Z frame #4: <unknown function> + 0x5a02023 (0x7f9f93510023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3091583Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9f935141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3092182Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9f9350b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3092577Z frame #7: <unknown function> + 0x79d655 (0x7f9fa49fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3092812Z frame #8: <unknown function> + 0xdc253 (0x7f9f5ba46253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3093033Z frame #9: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3093251Z frame #10: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3093255Z 
2025-07-17T11:07:26.3093352Z SIGABRT(6), PID: 1805791, Thread 1806152: 
2025-07-17T11:07:26.3093828Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3094042Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3094247Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3094452Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3094954Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3095168Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3095385Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3095390Z 
2025-07-17T11:07:26.3095481Z SIGABRT(6), PID: 1805794, Thread 1806844: 
2025-07-17T11:07:26.3096039Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3096725Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f6c60e5a08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3096953Z frame #2: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3097226Z frame #3: pthread_kill + 0x12c (0x7f6d527639fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3097412Z frame #4: raise + 0x16 (0x7f6d5270f476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3097656Z frame #5: abort + 0xd3 (0x7f6d526f57f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3097914Z frame #6: <unknown function> + 0x972cc (0x7f6bce9e32cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3098149Z frame #7: <unknown function> + 0x95f47 (0x7f6bce9e1f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3098382Z frame #8: <unknown function> + 0x34b71 (0x7f6bce980b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3098592Z frame #9: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3098813Z frame #10: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3098817Z 
2025-07-17T11:07:26.3098914Z SIGABRT(6), PID: 1805791, Thread 1806155: 
2025-07-17T11:07:26.3099393Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3099608Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3099823Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3100024Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3100550Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3100768Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3100993Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3100997Z 
2025-07-17T11:07:26.3101091Z SIGABRT(6), PID: 1805794, Thread 1809572: 
2025-07-17T11:07:26.3101577Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3101790Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3101974Z frame #2: __poll + 0x4f (0x7f6d527e5bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3102387Z frame #3: <unknown function> + 0x489f (0x7f6d522f189f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.3102634Z frame #4: <unknown function> + 0x1483bc (0x5631ff1903bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3102899Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3103141Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3103388Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3103628Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3103882Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5631ff178a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3104126Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3104382Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5631ff176c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3104706Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3105011Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3105254Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3105560Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3105804Z frame #16: <unknown function> + 0x147734 (0x5631ff18f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3106095Z frame #17: <unknown function> + 0x21ca82 (0x5631ff264a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3106327Z frame #18: <unknown function> + 0x21c9e4 (0x5631ff2649e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3106542Z frame #19: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3106766Z frame #20: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3106770Z 
2025-07-17T11:07:26.3106866Z SIGABRT(6), PID: 1805795, Thread 1805795: 
2025-07-17T11:07:26.3107362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3107581Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3107822Z frame #2: <unknown function> + 0x77445 (0x7f8e5fac6445 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3108058Z frame #3: <unknown function> + 0x76c8e (0x7f8e5fac5c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3108299Z frame #4: <unknown function> + 0x6ad69 (0x7f8e5fab9d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3108520Z frame #5: <unknown function> + 0x3f54db (0x7f8ef071e4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3108745Z frame #6: <unknown function> + 0x3e11d8 (0x7f8ef070a1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3108960Z frame #7: <unknown function> + 0xb2f86 (0x7f8ef03dbf86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3109178Z frame #8: <unknown function> + 0x8e501 (0x7f8ef03b7501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3109568Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f8f1b1be7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.3109962Z frame #10: <unknown function> + 0xd307d9 (0x7f8f397117d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3110191Z frame #11: <unknown function> + 0x139902 (0x5579e3066902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3110452Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5579e305da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3110690Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3110939Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5579e305da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3111168Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3111424Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3111656Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3111902Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3112125Z frame #19: <unknown function> + 0x1474b2 (0x5579e30744b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3112347Z frame #20: PyObject_Call + 0xbc (0x5579e3074e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3112573Z frame #21: <unknown function> + 0x1ff69d (0x5579e312c69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3112888Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5579e3062023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3113205Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5579e305dcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3113444Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3113745Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5579e30593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3113981Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3114287Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5579e30593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3114519Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3114764Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5579e305da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3115030Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5579e30614b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3115256Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5579e3072ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3115485Z frame #32: <unknown function> + 0x205949 (0x5579e3132949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3115713Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5579e3062023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3115965Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5579e305dcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3116196Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3116447Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5579e30593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3116674Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3116923Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5579e30593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3117150Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3117395Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5579e30593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3117623Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3117877Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5579e305a3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3118105Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3118507Z frame #44: <unknown function> + 0x7ab3f7 (0x7f8f3918c3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3118736Z frame #45: <unknown function> + 0x13bc88 (0x5579e3068c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3118951Z frame #46: PyObject_Call + 0x20f (0x5579e3074f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3119200Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5579e305e98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3119428Z frame #48: <unknown function> + 0x147608 (0x5579e3074608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3119675Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5579e305bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3119909Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3120153Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5579e30593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3120384Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3120626Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5579e305a3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3120940Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3121248Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5579e30593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3121481Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3121782Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5579e305bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3122017Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3122351Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5579e305bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3122577Z frame #60: <unknown function> + 0x147608 (0x5579e3074608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3122832Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5579e305bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3123058Z frame #62: <unknown function> + 0x147608 (0x5579e3074608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3123062Z 
2025-07-17T11:07:26.3123158Z SIGABRT(6), PID: 1805791, Thread 1806159: 
2025-07-17T11:07:26.3123648Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3123868Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3124094Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3124301Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3124821Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3125034Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3125251Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3125256Z 
2025-07-17T11:07:26.3125350Z SIGABRT(6), PID: 1805791, Thread 1806162: 
2025-07-17T11:07:26.3125836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3126041Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3126253Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3126460Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3126973Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3127185Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3127405Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3127409Z 
2025-07-17T11:07:26.3127507Z SIGABRT(6), PID: 1805795, Thread 1806194: 
2025-07-17T11:07:26.3127980Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3128194Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3128397Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3128601Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3129179Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3129458Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3129683Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3129687Z 
2025-07-17T11:07:26.3129842Z SIGABRT(6), PID: 1805791, Thread 1806165: 
2025-07-17T11:07:26.3130317Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3130594Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3130801Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3131004Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3131507Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3131722Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3131933Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3131938Z 
2025-07-17T11:07:26.3132037Z SIGABRT(6), PID: 1805795, Thread 1806200: 
2025-07-17T11:07:26.3132519Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3132732Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3132936Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3133144Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3133647Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3133861Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3134071Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3134076Z 
2025-07-17T11:07:26.3134175Z SIGABRT(6), PID: 1805794, Thread 1809573: 
2025-07-17T11:07:26.3134646Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3134861Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3135064Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3135292Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6d527612dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3135743Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f6c8cba2048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3136253Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f6c8cba4bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3136491Z frame #6: <unknown function> + 0xdc253 (0x7f6c5f0c6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3136706Z frame #7: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3136918Z frame #8: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3137015Z SIGABRT(6), PID: 1805791, Thread 1806169: 
2025-07-17T11:07:26.3137582Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3137878Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3138087Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3138366Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3138866Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3139162Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3139373Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3139384Z 
2025-07-17T11:07:26.3139475Z SIGABRT(6), PID: 1805795, Thread 1806206: 
2025-07-17T11:07:26.3139953Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3140165Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3140376Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3140580Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3141087Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3141294Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3141508Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3141604Z SIGABRT(6), PID: 1805790, Thread 1810660: 
2025-07-17T11:07:26.3142088Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3142293Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3142504Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3142705Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3143101Z frame #4: <unknown function> + 0x5a02023 (0x7f9f93510023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3143676Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9f935141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3144281Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9f9350b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3144668Z frame #7: <unknown function> + 0x79d655 (0x7f9fa49fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3144911Z frame #8: <unknown function> + 0xdc253 (0x7f9f5ba46253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3145126Z frame #9: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3145351Z frame #10: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3145444Z SIGABRT(6), PID: 1805793, Thread 1805793: 
2025-07-17T11:07:26.3145925Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3146215Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3146525Z frame #2: <unknown function> + 0x7743a (0x7f618754b43a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3146761Z frame #3: <unknown function> + 0x76c8e (0x7f618754ac8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3147056Z frame #4: <unknown function> + 0x6ad69 (0x7f618753ed69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3147275Z frame #5: <unknown function> + 0x3f54db (0x7f62181a34db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3147564Z frame #6: <unknown function> + 0x3e11d8 (0x7f621818f1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3147778Z frame #7: <unknown function> + 0xb2f86 (0x7f6217e60f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3147993Z frame #8: <unknown function> + 0x8e501 (0x7f6217e3c501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3148381Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f6242c437ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.3148772Z frame #10: <unknown function> + 0xd307d9 (0x7f62611967d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3149000Z frame #11: <unknown function> + 0x139902 (0x556ed0d2e902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3149264Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x556ed0d25a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3149497Z frame #13: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3149749Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x556ed0d25a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3149976Z frame #15: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3150226Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3150452Z frame #17: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3150698Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3150921Z frame #19: <unknown function> + 0x1474b2 (0x556ed0d3c4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3151132Z frame #20: PyObject_Call + 0xbc (0x556ed0d3ce4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3151357Z frame #21: <unknown function> + 0x1ff69d (0x556ed0df469d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3151587Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x556ed0d2a023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3151830Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x556ed0d25cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3152062Z frame #24: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3152304Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x556ed0d213bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3152543Z frame #26: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3152785Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x556ed0d213bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3153027Z frame #28: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3153270Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x556ed0d25a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3153535Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x556ed0d294b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3153761Z frame #31: _PyObject_Call_Prepend + 0x69 (0x556ed0d3aab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3153988Z frame #32: <unknown function> + 0x205949 (0x556ed0dfa949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3154294Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x556ed0d2a023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3154607Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x556ed0d25cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3154837Z frame #35: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3155164Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x556ed0d213bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3155392Z frame #37: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3155702Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x556ed0d213bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3155929Z frame #39: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3156170Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x556ed0d213bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3156398Z frame #41: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3156646Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x556ed0d223f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3156872Z frame #43: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3157271Z frame #44: <unknown function> + 0x7ab3f7 (0x7f6260c113f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3157495Z frame #45: <unknown function> + 0x13bc88 (0x556ed0d30c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3157712Z frame #46: PyObject_Call + 0x20f (0x556ed0d3cf9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3157959Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x556ed0d2698d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3158184Z frame #48: <unknown function> + 0x147608 (0x556ed0d3c608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3158426Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x556ed0d23c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3158659Z frame #50: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3158897Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x556ed0d213bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3159127Z frame #52: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3159373Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x556ed0d223f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3159607Z frame #54: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3159850Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x556ed0d213bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3160081Z frame #56: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3160326Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x556ed0d23c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3160559Z frame #58: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3160798Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x556ed0d23c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3161025Z frame #60: <unknown function> + 0x147608 (0x556ed0d3c608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3161270Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x556ed0d23c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3161496Z frame #62: <unknown function> + 0x147608 (0x556ed0d3c608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3161500Z 
2025-07-17T11:07:26.3161504Z 
2025-07-17T11:07:26.3161507Z 
2025-07-17T11:07:26.3161510Z 
2025-07-17T11:07:26.3161605Z SIGABRT(6), PID: 1805791, Thread 1806173: 
2025-07-17T11:07:26.3162173Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3162448Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3162668Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3162931Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3163443Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3163717Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3163934Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3163940Z 
2025-07-17T11:07:26.3164036Z SIGABRT(6), PID: 1805795, Thread 1806211: 
2025-07-17T11:07:26.3164526Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3164746Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3164956Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3165162Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3165668Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3165890Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3166102Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3166109Z 
2025-07-17T11:07:26.3166206Z SIGABRT(6), PID: 1805793, Thread 1805950: 
2025-07-17T11:07:26.3166678Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3166890Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3167100Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3167307Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3167802Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3168019Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3168232Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3168237Z 
2025-07-17T11:07:26.3168333Z SIGABRT(6), PID: 1805794, Thread 1809574: 
2025-07-17T11:07:26.3168804Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3169021Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3169227Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3169460Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6d527612dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3169935Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f6c8cb9c741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3170261Z frame #5: <unknown function> + 0xdc253 (0x7f6c5f0c6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3170471Z frame #6: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3170757Z frame #7: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3170762Z 
2025-07-17T11:07:26.3170859Z SIGABRT(6), PID: 1805791, Thread 1806177: 
2025-07-17T11:07:26.3171425Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3171724Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3171946Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3172152Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3172683Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3172898Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3173110Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3173114Z 
2025-07-17T11:07:26.3173209Z SIGABRT(6), PID: 1805795, Thread 1806214: 
2025-07-17T11:07:26.3173701Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3173913Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3174122Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3174324Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3174835Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3175047Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3175268Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3175272Z 
2025-07-17T11:07:26.3175364Z SIGABRT(6), PID: 1805793, Thread 1805953: 
2025-07-17T11:07:26.3175837Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3176125Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3176335Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3176542Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3177050Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3177270Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3177481Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3177486Z 
2025-07-17T11:07:26.3177588Z SIGABRT(6), PID: 1805794, Thread 1809577: 
2025-07-17T11:07:26.3178061Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3178278Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3178450Z frame #2: ioctl + 0x3f (0x7f6d527e794f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3178800Z frame #3: <unknown function> + 0x13eef0 (0x7f6bcea8aef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3179112Z frame #4: <unknown function> + 0x1378c4 (0x7f6bcea838c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3179358Z frame #5: <unknown function> + 0x95b2b (0x7f6bce9e1b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3179661Z frame #6: <unknown function> + 0x34b71 (0x7f6bce980b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3179882Z frame #7: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3180171Z frame #8: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3180177Z 
2025-07-17T11:07:26.3180276Z SIGABRT(6), PID: 1805791, Thread 1806181: 
2025-07-17T11:07:26.3180752Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3180970Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3181180Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3181393Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3181895Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3182115Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3182329Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3182332Z 
2025-07-17T11:07:26.3182429Z SIGABRT(6), PID: 1805793, Thread 1805957: 
2025-07-17T11:07:26.3182904Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3183118Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3183323Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3183533Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3184040Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3184254Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3184465Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3184469Z 
2025-07-17T11:07:26.3184568Z SIGABRT(6), PID: 1805795, Thread 1806218: 
2025-07-17T11:07:26.3185037Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3185252Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3185457Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3185664Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3186167Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3186383Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3186593Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3186676Z 
2025-07-17T11:07:26.3186777Z SIGABRT(6), PID: 1805791, Thread 1806185: 
2025-07-17T11:07:26.3187340Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3187556Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3187835Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3188041Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3188617Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3188833Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3189056Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3189060Z 
2025-07-17T11:07:26.3189151Z SIGABRT(6), PID: 1805794, Thread 1810598: 
2025-07-17T11:07:26.3189633Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3189841Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3190020Z frame #2: __poll + 0x4f (0x7f6d527e5bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3190239Z frame #3: <unknown function> + 0xd4540 (0x7f6c16bfa540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3190453Z frame #4: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3190665Z frame #5: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3190669Z 
2025-07-17T11:07:26.3190770Z SIGABRT(6), PID: 1805793, Thread 1805961: 
2025-07-17T11:07:26.3198534Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3198828Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3199054Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3199275Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3199803Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3200022Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3200237Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3200342Z SIGABRT(6), PID: 1805795, Thread 1806221: 
2025-07-17T11:07:26.3200843Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3201060Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3201266Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3201475Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3201994Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3202211Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3202424Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3202549Z 
2025-07-17T11:07:26.3202554Z 
2025-07-17T11:07:26.3202657Z SIGABRT(6), PID: 1805794, Thread 1810608: 
2025-07-17T11:07:26.3203205Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3203429Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3203665Z frame #2: __poll + 0x4f (0x7f6d527e5bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3203960Z frame #3: <unknown function> + 0xd65a2 (0x7f6c16bfc5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3204172Z frame #4: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3204387Z frame #5: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3204392Z 
2025-07-17T11:07:26.3204487Z SIGABRT(6), PID: 1805791, Thread 1806189: 
2025-07-17T11:07:26.3204976Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3205193Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3205411Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3205620Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3206144Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3206352Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3206562Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3206568Z 
2025-07-17T11:07:26.3206661Z SIGABRT(6), PID: 1805794, Thread 1810615: 
2025-07-17T11:07:26.3207142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3207351Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3207551Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3207753Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3207971Z frame #4: <unknown function> + 0xd1e7f (0x7f6c16bf7e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3208174Z frame #5: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3208383Z frame #6: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3208390Z 
2025-07-17T11:07:26.3208484Z SIGABRT(6), PID: 1805791, Thread 1806192: 
2025-07-17T11:07:26.3208957Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3209164Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3209366Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3209563Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3210068Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3210277Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3210568Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3210661Z SIGABRT(6), PID: 1805795, Thread 1806223: 
2025-07-17T11:07:26.3211192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3211405Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3211667Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3211932Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3212436Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3212644Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3212854Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3212859Z 
2025-07-17T11:07:26.3212862Z 
2025-07-17T11:07:26.3212955Z SIGABRT(6), PID: 1805793, Thread 1805963: 
2025-07-17T11:07:26.3213419Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3213628Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3213830Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3214031Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3214528Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3214740Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3214954Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3214958Z 
2025-07-17T11:07:26.3215052Z SIGABRT(6), PID: 1805794, Thread 1810626: 
2025-07-17T11:07:26.3215520Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3215729Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3216029Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3216244Z frame #3: <unknown function> + 0x9ca53 (0x7f6d52769a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3216506Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5631ff15cbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3216741Z frame #5: <unknown function> + 0x1d314f (0x5631ff21b14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3216964Z frame #6: <unknown function> + 0x1d2f6b (0x5631ff21af6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3217186Z frame #7: <unknown function> + 0x6df36 (0x5631ff0b5f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3217426Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3217654Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3217896Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3218128Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3218361Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3218688Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3218924Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3219219Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3219453Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3219746Z frame #17: <unknown function> + 0x147734 (0x5631ff18f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3220036Z frame #18: <unknown function> + 0x21ca82 (0x5631ff264a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3220256Z frame #19: <unknown function> + 0x21c9e4 (0x5631ff2649e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3220465Z frame #20: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3220684Z frame #21: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3220689Z 
2025-07-17T11:07:26.3220780Z SIGABRT(6), PID: 1805791, Thread 1806196: 
2025-07-17T11:07:26.3221263Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3221476Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3221684Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3221888Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3222395Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3222601Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3222813Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3222817Z 
2025-07-17T11:07:26.3222915Z SIGABRT(6), PID: 1805790, Thread 1810661: 
2025-07-17T11:07:26.3223385Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f5d7d9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3223595Z frame #1: <unknown function> + 0x42520 (0x7fa04f08f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3223802Z frame #2: <unknown function> + 0x91117 (0x7fa04f0de117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3224001Z frame #3: pthread_cond_wait + 0x211 (0x7fa04f0e0a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3224383Z frame #4: <unknown function> + 0x5a02023 (0x7f9f93510023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3224956Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9f935141ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3225554Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9f9350b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3225946Z frame #7: <unknown function> + 0x79d655 (0x7f9fa49fb655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3226182Z frame #8: <unknown function> + 0xdc253 (0x7f9f5ba46253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3226397Z frame #9: <unknown function> + 0x94ac3 (0x7fa04f0e1ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3226614Z frame #10: <unknown function> + 0x126850 (0x7fa04f173850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3226716Z 
2025-07-17T11:07:26.3226818Z SIGABRT(6), PID: 1805793, Thread 1805968: 
2025-07-17T11:07:26.3227355Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3227574Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3227780Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3228044Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3228615Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3228828Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3229036Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3229042Z 
2025-07-17T11:07:26.3229136Z SIGABRT(6), PID: 1805795, Thread 1806225: 
2025-07-17T11:07:26.3229612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3229823Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3230025Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3230224Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3230724Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3230933Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3231141Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3231146Z 
2025-07-17T11:07:26.3231241Z SIGABRT(6), PID: 1805791, Thread 1806201: 
2025-07-17T11:07:26.3231707Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3231914Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3232115Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3232317Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3232820Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3233029Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3233237Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3233244Z 
2025-07-17T11:07:26.3233335Z SIGABRT(6), PID: 1805793, Thread 1805973: 
2025-07-17T11:07:26.3233808Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3234011Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3234220Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3234415Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3234913Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3235197Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3235473Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3235566Z SIGABRT(6), PID: 1805794, Thread 1810645: 
2025-07-17T11:07:26.3236097Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3236305Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3236539Z frame #2: read + 0x4c (0x7f6d527e181c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3236719Z frame #3: _Py_read + 0x4c (0x5631ff222a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3236946Z frame #4: <unknown function> + 0x208c3b (0x5631ff250c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3237169Z frame #5: <unknown function> + 0x14648d (0x5631ff18e48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3237418Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5631ff19d215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3237638Z frame #7: <unknown function> + 0x208ae7 (0x5631ff250ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3237856Z frame #8: <unknown function> + 0x208472 (0x5631ff250472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3238074Z frame #9: <unknown function> + 0x1da6ee (0x5631ff2226ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3238303Z frame #10: <unknown function> + 0x1483bc (0x5631ff1903bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3238545Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3238771Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3239006Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5631ff1743bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3239230Z frame #14: <unknown function> + 0x147734 (0x5631ff18f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3239478Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5631ff176c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3239704Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3239941Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3240167Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5631ff1840ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3240397Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5631ff1747b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3240618Z frame #20: <unknown function> + 0x147734 (0x5631ff18f734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3240836Z frame #21: <unknown function> + 0x21ca82 (0x5631ff264a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3241060Z frame #22: <unknown function> + 0x21c9e4 (0x5631ff2649e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3241270Z frame #23: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3241481Z frame #24: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3241485Z 
2025-07-17T11:07:26.3241491Z 
2025-07-17T11:07:26.3241583Z SIGABRT(6), PID: 1805795, Thread 1806227: 
2025-07-17T11:07:26.3242063Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3242273Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3242481Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3242764Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3243338Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3243549Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3243822Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3243828Z 
2025-07-17T11:07:26.3243920Z SIGABRT(6), PID: 1805791, Thread 1806205: 
2025-07-17T11:07:26.3244455Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3244665Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3244868Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3245068Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3245569Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3245776Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3245984Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3245991Z 
2025-07-17T11:07:26.3246078Z SIGABRT(6), PID: 1805793, Thread 1805977: 
2025-07-17T11:07:26.3246551Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3246753Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3246960Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3247158Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3247666Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3247872Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3248080Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3248086Z 
2025-07-17T11:07:26.3248174Z SIGABRT(6), PID: 1805795, Thread 1806228: 
2025-07-17T11:07:26.3248640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3248843Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3249050Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3249250Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3249751Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3249954Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3250166Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3250171Z 
2025-07-17T11:07:26.3250262Z SIGABRT(6), PID: 1805791, Thread 1806210: 
2025-07-17T11:07:26.3250730Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3251015Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3251283Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3251487Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3252042Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3252330Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3252546Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3252550Z 
2025-07-17T11:07:26.3252642Z SIGABRT(6), PID: 1805793, Thread 1805982: 
2025-07-17T11:07:26.3253109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3253318Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3253520Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3253720Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3254216Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3254423Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3254629Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3254633Z 
2025-07-17T11:07:26.3254722Z SIGABRT(6), PID: 1805791, Thread 1806215: 
2025-07-17T11:07:26.3255191Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3255401Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3255601Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3255800Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3256370Z frame #4: <unknown function> + 0x3475ab (0x7fa1a602c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3256582Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3256789Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3256795Z 
2025-07-17T11:07:26.3256886Z SIGABRT(6), PID: 1805795, Thread 1806230: 
2025-07-17T11:07:26.3257353Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3257562Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3257764Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3257964Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3258463Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3258672Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3258878Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3259077Z SIGABRT(6), PID: 1805793, Thread 1805986: 
2025-07-17T11:07:26.3259628Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3259837Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3260139Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3260340Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3260910Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3261117Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3261324Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3261335Z 
2025-07-17T11:07:26.3261338Z 
2025-07-17T11:07:26.3261427Z SIGABRT(6), PID: 1805791, Thread 1806849: 
2025-07-17T11:07:26.3261891Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3262344Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fa0b788a08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3262553Z frame #2: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3262746Z frame #3: pthread_kill + 0x12c (0x7fa1a91969fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3262916Z frame #4: raise + 0x16 (0x7fa1a9142476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3263080Z frame #5: abort + 0xd3 (0x7fa1a91287f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3263320Z frame #6: <unknown function> + 0x972cc (0x7fa0254132cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3263548Z frame #7: <unknown function> + 0x95f47 (0x7fa025411f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3263778Z frame #8: <unknown function> + 0x34b71 (0x7fa0253b0b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3263988Z frame #9: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3264202Z frame #10: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3264208Z 
2025-07-17T11:07:26.3264295Z SIGABRT(6), PID: 1805793, Thread 1805991: 
2025-07-17T11:07:26.3264774Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3264977Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3265186Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3265386Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3265882Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3266087Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3266296Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3266301Z 
2025-07-17T11:07:26.3266389Z SIGABRT(6), PID: 1805795, Thread 1806232: 
2025-07-17T11:07:26.3266860Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3267144Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3267348Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3267609Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3268178Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3268385Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3268657Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3268663Z 
2025-07-17T11:07:26.3268751Z SIGABRT(6), PID: 1805791, Thread 1809774: 
2025-07-17T11:07:26.3269217Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3269422Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3269592Z frame #2: __poll + 0x4f (0x7fa1a9218bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3270002Z frame #3: <unknown function> + 0x489f (0x7fa1a8d2489f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.3270235Z frame #4: <unknown function> + 0x1483bc (0x5561bb1923bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3270484Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3270712Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3270944Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3271170Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3271413Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5561bb17aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3271650Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3271893Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5561bb178c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3272120Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3272354Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3272589Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3272819Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3273039Z frame #16: <unknown function> + 0x147734 (0x5561bb191734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3273258Z frame #17: <unknown function> + 0x21ca82 (0x5561bb266a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3273482Z frame #18: <unknown function> + 0x21c9e4 (0x5561bb2669e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3273692Z frame #19: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3273903Z frame #20: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3273908Z 
2025-07-17T11:07:26.3274000Z SIGABRT(6), PID: 1805793, Thread 1805996: 
2025-07-17T11:07:26.3274479Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3274686Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3274971Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3275169Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3275730Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3275999Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3276214Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3276279Z 
2025-07-17T11:07:26.3276371Z SIGABRT(6), PID: 1805795, Thread 1806234: 
2025-07-17T11:07:26.3276902Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3277126Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3277332Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3277533Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3278035Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3278243Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3278453Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3278457Z 
2025-07-17T11:07:26.3278546Z SIGABRT(6), PID: 1805795, Thread 1806236: 
2025-07-17T11:07:26.3279011Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3279221Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3279422Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3279622Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3280117Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3280326Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3280534Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3280628Z SIGABRT(6), PID: 1805793, Thread 1806001: 
2025-07-17T11:07:26.3281096Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3281321Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3281524Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3281722Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3282220Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3282432Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3282640Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3282644Z 
2025-07-17T11:07:26.3282647Z 
2025-07-17T11:07:26.3282734Z SIGABRT(6), PID: 1805794, Thread 1810686: 
2025-07-17T11:07:26.3283200Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3283543Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3283747Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3283944Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3284405Z frame #4: <unknown function> + 0x5a02023 (0x7f6c96b90023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3285030Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c96b941ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3285628Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c96b8b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3286020Z frame #7: <unknown function> + 0x79d655 (0x7f6ca8077655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3286253Z frame #8: <unknown function> + 0xdc253 (0x7f6c5f0c6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3286463Z frame #9: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3286674Z frame #10: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3286684Z 
2025-07-17T11:07:26.3286771Z SIGABRT(6), PID: 1805795, Thread 1806238: 
2025-07-17T11:07:26.3287247Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3287451Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3287654Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3287858Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3288363Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3288569Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3288782Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3288786Z 
2025-07-17T11:07:26.3288874Z SIGABRT(6), PID: 1805793, Thread 1806006: 
2025-07-17T11:07:26.3289349Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3289555Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3289765Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3289964Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3290470Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3290679Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3290896Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3290900Z 
2025-07-17T11:07:26.3290989Z SIGABRT(6), PID: 1805791, Thread 1809776: 
2025-07-17T11:07:26.3291462Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3291748Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3292017Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3292244Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa1a91942dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3292757Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fa0e35d2048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3293243Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fa0e35d4bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3293482Z frame #6: <unknown function> + 0xdc253 (0x7fa0b5af6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3293692Z frame #7: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3293906Z frame #8: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3293911Z 
2025-07-17T11:07:26.3294002Z SIGABRT(6), PID: 1805795, Thread 1806240: 
2025-07-17T11:07:26.3294482Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3294689Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3294896Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3295091Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3295590Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3295802Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3296081Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3296089Z 
2025-07-17T11:07:26.3296177Z SIGABRT(6), PID: 1805793, Thread 1806012: 
2025-07-17T11:07:26.3296655Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3296861Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3297068Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3297261Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3297763Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3297969Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3298186Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3298190Z 
2025-07-17T11:07:26.3298284Z SIGABRT(6), PID: 1805793, Thread 1806018: 
2025-07-17T11:07:26.3298754Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3298967Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3299177Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3299380Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3299974Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3300263Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3300477Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3300575Z SIGABRT(6), PID: 1805795, Thread 1806242: 
2025-07-17T11:07:26.3301113Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3301390Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3301603Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3301812Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3302317Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3302533Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3302742Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3302748Z 
2025-07-17T11:07:26.3302751Z 
2025-07-17T11:07:26.3302852Z SIGABRT(6), PID: 1805791, Thread 1809777: 
2025-07-17T11:07:26.3303333Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3303550Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3303753Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3303982Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa1a91942dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3304457Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fa0e35cc741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3304689Z frame #5: <unknown function> + 0xdc253 (0x7fa0b5af6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3304897Z frame #6: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3305114Z frame #7: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3305120Z 
2025-07-17T11:07:26.3305209Z SIGABRT(6), PID: 1805793, Thread 1806023: 
2025-07-17T11:07:26.3305686Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3305889Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3306100Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3306304Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3306806Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3307011Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3307225Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3307229Z 
2025-07-17T11:07:26.3307317Z SIGABRT(6), PID: 1805791, Thread 1809829: 
2025-07-17T11:07:26.3307785Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3308066Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3308309Z frame #2: ioctl + 0x3f (0x7fa1a921a94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3308550Z frame #3: <unknown function> + 0x13eef0 (0x7fa0254baef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3308787Z frame #4: <unknown function> + 0x1378c4 (0x7fa0254b38c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3309084Z frame #5: <unknown function> + 0x95b2b (0x7fa025411b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3309371Z frame #6: <unknown function> + 0x34b71 (0x7fa0253b0b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3309579Z frame #7: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3309788Z frame #8: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3309794Z 
2025-07-17T11:07:26.3309888Z SIGABRT(6), PID: 1805795, Thread 1806244: 
2025-07-17T11:07:26.3310367Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3310576Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3310781Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3310984Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3311482Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3311693Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3311898Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3311906Z 
2025-07-17T11:07:26.3311996Z SIGABRT(6), PID: 1805793, Thread 1806028: 
2025-07-17T11:07:26.3312461Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3312674Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3312876Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3313076Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3313570Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3313782Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3313992Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3313997Z 
2025-07-17T11:07:26.3314090Z SIGABRT(6), PID: 1805795, Thread 1806246: 
2025-07-17T11:07:26.3314554Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3314760Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3314961Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3315160Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3315659Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3315956Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3316166Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3316170Z 
2025-07-17T11:07:26.3316340Z SIGABRT(6), PID: 1805791, Thread 1810601: 
2025-07-17T11:07:26.3316810Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3317079Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3317304Z frame #2: __poll + 0x4f (0x7fa1a9218bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3317522Z frame #3: <unknown function> + 0xd4540 (0x7fa06d62a540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3317725Z frame #4: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3317935Z frame #5: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3317941Z 
2025-07-17T11:07:26.3318027Z SIGABRT(6), PID: 1805793, Thread 1806033: 
2025-07-17T11:07:26.3318499Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3318706Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3318915Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3319113Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3319608Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3319813Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3320021Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3320026Z 
2025-07-17T11:07:26.3320114Z SIGABRT(6), PID: 1805794, Thread 1810687: 
2025-07-17T11:07:26.3320582Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3320793Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3320997Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3321192Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3321570Z frame #4: <unknown function> + 0x5a02023 (0x7f6c96b90023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3322141Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c96b941ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3322737Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c96b8b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3323132Z frame #7: <unknown function> + 0x79d655 (0x7f6ca8077655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3323362Z frame #8: <unknown function> + 0xdc253 (0x7f6c5f0c6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3323570Z frame #9: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3323785Z frame #10: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3323790Z 
2025-07-17T11:07:26.3323962Z SIGABRT(6), PID: 1805791, Thread 1810612: 
2025-07-17T11:07:26.3324432Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3324703Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3324877Z frame #2: __poll + 0x4f (0x7fa1a9218bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3325158Z frame #3: <unknown function> + 0xd65a2 (0x7fa06d62c5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3325368Z frame #4: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3325641Z frame #5: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3325646Z 
2025-07-17T11:07:26.3325734Z SIGABRT(6), PID: 1805795, Thread 1806248: 
2025-07-17T11:07:26.3326208Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3326415Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3326622Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3326817Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3327316Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3327524Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3327734Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3327737Z 
2025-07-17T11:07:26.3327824Z SIGABRT(6), PID: 1805791, Thread 1810616: 
2025-07-17T11:07:26.3328293Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3328502Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3328707Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3328905Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3329126Z frame #4: <unknown function> + 0xd1e7f (0x7fa06d627e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3329330Z frame #5: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3329539Z frame #6: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3329543Z 
2025-07-17T11:07:26.3329630Z SIGABRT(6), PID: 1805793, Thread 1806036: 
2025-07-17T11:07:26.3330100Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3330308Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3330514Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3330705Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3331199Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3331404Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3331612Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3331616Z 
2025-07-17T11:07:26.3331781Z SIGABRT(6), PID: 1805795, Thread 1806250: 
2025-07-17T11:07:26.3332249Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3332517Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3332731Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3332990Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3333549Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3333757Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3333965Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3333970Z 
2025-07-17T11:07:26.3334058Z SIGABRT(6), PID: 1805791, Thread 1810628: 
2025-07-17T11:07:26.3334526Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3334732Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3334935Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3335142Z frame #3: <unknown function> + 0x9ca53 (0x7fa1a919ca53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3335397Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5561bb15ebc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3335631Z frame #5: <unknown function> + 0x1d314f (0x5561bb21d14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3335853Z frame #6: <unknown function> + 0x1d2f6b (0x5561bb21cf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3336136Z frame #7: <unknown function> + 0x6df36 (0x5561bb0b7f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3336380Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3336604Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3336846Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3337075Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3337314Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3337538Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3337770Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3338000Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3338232Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3338449Z frame #17: <unknown function> + 0x147734 (0x5561bb191734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3338665Z frame #18: <unknown function> + 0x21ca82 (0x5561bb266a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3338886Z frame #19: <unknown function> + 0x21c9e4 (0x5561bb2669e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3339097Z frame #20: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3339308Z frame #21: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3339395Z SIGABRT(6), PID: 1805793, Thread 1806040: 
2025-07-17T11:07:26.3339865Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3340168Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3340460Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3340665Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3341232Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3341511Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3341719Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3341723Z 
2025-07-17T11:07:26.3341727Z 
2025-07-17T11:07:26.3341816Z SIGABRT(6), PID: 1805795, Thread 1806252: 
2025-07-17T11:07:26.3342285Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3342489Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3342694Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3342892Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3343405Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3343613Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3343821Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3343827Z 
2025-07-17T11:07:26.3343913Z SIGABRT(6), PID: 1805795, Thread 1806253: 
2025-07-17T11:07:26.3344385Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3344588Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3344800Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3344995Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3345494Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3345698Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3345911Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3345915Z 
2025-07-17T11:07:26.3346004Z SIGABRT(6), PID: 1805793, Thread 1806044: 
2025-07-17T11:07:26.3346469Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3346673Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3346877Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3347073Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3347566Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3347774Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3348078Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3348085Z 
2025-07-17T11:07:26.3348239Z SIGABRT(6), PID: 1805791, Thread 1810644: 
2025-07-17T11:07:26.3348706Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3348970Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3349137Z frame #2: read + 0x4c (0x7fa1a921481c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3349385Z frame #3: _Py_read + 0x4c (0x5561bb224a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3349609Z frame #4: <unknown function> + 0x208c3b (0x5561bb252c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3349830Z frame #5: <unknown function> + 0x14648d (0x5561bb19048d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3350078Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5561bb19f215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3350299Z frame #7: <unknown function> + 0x208ae7 (0x5561bb252ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3350512Z frame #8: <unknown function> + 0x208472 (0x5561bb252472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3350737Z frame #9: <unknown function> + 0x1da6ee (0x5561bb2246ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3350958Z frame #10: <unknown function> + 0x1483bc (0x5561bb1923bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3351204Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3351430Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3351665Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5561bb1763bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3351884Z frame #14: <unknown function> + 0x147734 (0x5561bb191734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3352137Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5561bb178c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3352359Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3352597Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3352828Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5561bb1860ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3353064Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5561bb1767b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3353280Z frame #20: <unknown function> + 0x147734 (0x5561bb191734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3353500Z frame #21: <unknown function> + 0x21ca82 (0x5561bb266a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3353721Z frame #22: <unknown function> + 0x21c9e4 (0x5561bb2669e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3353937Z frame #23: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3354148Z frame #24: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3354152Z 
2025-07-17T11:07:26.3354244Z SIGABRT(6), PID: 1805795, Thread 1806255: 
2025-07-17T11:07:26.3354713Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3354923Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3355125Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3355324Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3355903Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3356176Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3356388Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3356392Z 
2025-07-17T11:07:26.3356548Z SIGABRT(6), PID: 1805793, Thread 1806048: 
2025-07-17T11:07:26.3357020Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3357289Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3357491Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3357692Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3358184Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3358392Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3358599Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3358605Z 
2025-07-17T11:07:26.3358699Z SIGABRT(6), PID: 1805795, Thread 1806258: 
2025-07-17T11:07:26.3359165Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3359375Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3359574Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3359772Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3360265Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3360475Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3360683Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3360687Z 
2025-07-17T11:07:26.3360780Z SIGABRT(6), PID: 1805793, Thread 1806052: 
2025-07-17T11:07:26.3361245Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3361449Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3361651Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3361847Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3362339Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3362548Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3362754Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3362764Z 
2025-07-17T11:07:26.3362854Z SIGABRT(6), PID: 1805795, Thread 1806260: 
2025-07-17T11:07:26.3363326Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3363531Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3363816Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3364077Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3364586Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3364854Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3365125Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3365129Z 
2025-07-17T11:07:26.3365217Z SIGABRT(6), PID: 1805794, Thread 1810688: 
2025-07-17T11:07:26.3365691Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3365902Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3366108Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3366307Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3366693Z frame #4: <unknown function> + 0x5a02023 (0x7f6c96b90023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3367263Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c96b941ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3367864Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c96b8b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3368248Z frame #7: <unknown function> + 0x79d655 (0x7f6ca8077655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3368480Z frame #8: <unknown function> + 0xdc253 (0x7f6c5f0c6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3368691Z frame #9: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3368908Z frame #10: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3368912Z 
2025-07-17T11:07:26.3369005Z SIGABRT(6), PID: 1805793, Thread 1806056: 
2025-07-17T11:07:26.3369477Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3369683Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3369890Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3370088Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3370590Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3370795Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3371006Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3371012Z 
2025-07-17T11:07:26.3371100Z SIGABRT(6), PID: 1805795, Thread 1806261: 
2025-07-17T11:07:26.3371569Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3371773Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3372058Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3372256Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3372815Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3373089Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3373305Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3373390Z 
2025-07-17T11:07:26.3373481Z SIGABRT(6), PID: 1805793, Thread 1806060: 
2025-07-17T11:07:26.3373948Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3374155Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3374357Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3374559Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3375060Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3375267Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3375471Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3375477Z 
2025-07-17T11:07:26.3375568Z SIGABRT(6), PID: 1805795, Thread 1806263: 
2025-07-17T11:07:26.3376095Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3376305Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3376506Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3376702Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3377201Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3377411Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3377615Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3377620Z 
2025-07-17T11:07:26.3377709Z SIGABRT(6), PID: 1805793, Thread 1806065: 
2025-07-17T11:07:26.3378171Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3378379Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3378579Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3378778Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3379271Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3379485Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3379692Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3379696Z 
2025-07-17T11:07:26.3379787Z SIGABRT(6), PID: 1805795, Thread 1806265: 
2025-07-17T11:07:26.3380348Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3380629Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3380835Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3381111Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3381613Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3381898Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3382105Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3382111Z 
2025-07-17T11:07:26.3382203Z SIGABRT(6), PID: 1805793, Thread 1806069: 
2025-07-17T11:07:26.3382666Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3382876Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3383076Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3383283Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3383776Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3383986Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3384192Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3384289Z SIGABRT(6), PID: 1805791, Thread 1810693: 
2025-07-17T11:07:26.3384755Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3384963Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3385165Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3385362Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3385738Z frame #4: <unknown function> + 0x5a02023 (0x7fa0ed5c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3386298Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa0ed5c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3386899Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa0ed5bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3387282Z frame #7: <unknown function> + 0x79d655 (0x7fa0feaa7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3387518Z frame #8: <unknown function> + 0xdc253 (0x7fa0b5af6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3387726Z frame #9: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3387941Z frame #10: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3387946Z 
2025-07-17T11:07:26.3387950Z 
2025-07-17T11:07:26.3388037Z SIGABRT(6), PID: 1805795, Thread 1806267: 
2025-07-17T11:07:26.3388507Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3388854Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3389063Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3389261Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3389826Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3390094Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3390302Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3390306Z 
2025-07-17T11:07:26.3390392Z SIGABRT(6), PID: 1805793, Thread 1806072: 
2025-07-17T11:07:26.3390862Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3391065Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3391269Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3391469Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3391961Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3392168Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3392378Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3392384Z 
2025-07-17T11:07:26.3392472Z SIGABRT(6), PID: 1805795, Thread 1806269: 
2025-07-17T11:07:26.3392947Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3393149Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3393354Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3393547Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3394048Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3394255Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3394466Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3394472Z 
2025-07-17T11:07:26.3394562Z SIGABRT(6), PID: 1805794, Thread 1810689: 
2025-07-17T11:07:26.3395031Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3395238Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3395442Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3395640Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3396013Z frame #4: <unknown function> + 0x5a02023 (0x7f6c96b90023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3396572Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c96b941ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3397299Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c96b8b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3397740Z frame #7: <unknown function> + 0x79d655 (0x7f6ca8077655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3397975Z frame #8: <unknown function> + 0xdc253 (0x7f6c5f0c6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3398244Z frame #9: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3398455Z frame #10: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3398460Z 
2025-07-17T11:07:26.3398550Z SIGABRT(6), PID: 1805793, Thread 1806075: 
2025-07-17T11:07:26.3399024Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3399240Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3399440Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3399638Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3400132Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3400348Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3400554Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3400558Z 
2025-07-17T11:07:26.3400655Z SIGABRT(6), PID: 1805795, Thread 1806271: 
2025-07-17T11:07:26.3401125Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3401331Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3401531Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3401732Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3402227Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3402433Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3402640Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3402646Z 
2025-07-17T11:07:26.3402735Z SIGABRT(6), PID: 1805793, Thread 1806079: 
2025-07-17T11:07:26.3403202Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3403408Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3403614Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3403812Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3404304Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3404515Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3404809Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3404813Z 
2025-07-17T11:07:26.3404902Z SIGABRT(6), PID: 1805795, Thread 1806273: 
2025-07-17T11:07:26.3405442Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3405652Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3405936Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3406191Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3406696Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3406906Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3407118Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3407122Z 
2025-07-17T11:07:26.3407211Z SIGABRT(6), PID: 1805795, Thread 1806275: 
2025-07-17T11:07:26.3407687Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3407895Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3408108Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3408302Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3408814Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3409019Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3409228Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3409318Z SIGABRT(6), PID: 1805793, Thread 1806082: 
2025-07-17T11:07:26.3409787Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3410000Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3410209Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3410412Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3410909Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3411115Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3411326Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3411330Z 
2025-07-17T11:07:26.3411333Z 
2025-07-17T11:07:26.3411422Z SIGABRT(6), PID: 1805793, Thread 1806085: 
2025-07-17T11:07:26.3411892Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3412099Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3412303Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3412499Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3413088Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3413353Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3413575Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3413667Z SIGABRT(6), PID: 1805795, Thread 1806278: 
2025-07-17T11:07:26.3414197Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3414477Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3414681Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3414888Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3415406Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3415617Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3415841Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3415847Z 
2025-07-17T11:07:26.3415850Z 
2025-07-17T11:07:26.3416016Z SIGABRT(6), PID: 1805791, Thread 1810694: 
2025-07-17T11:07:26.3416493Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3416714Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3416914Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3417131Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3417508Z frame #4: <unknown function> + 0x5a02023 (0x7fa0ed5c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3418081Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa0ed5c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3418672Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa0ed5bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3419064Z frame #7: <unknown function> + 0x79d655 (0x7fa0feaa7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3419294Z frame #8: <unknown function> + 0xdc253 (0x7fa0b5af6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3419512Z frame #9: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3419727Z frame #10: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3419731Z 
2025-07-17T11:07:26.3419826Z SIGABRT(6), PID: 1805795, Thread 1806280: 
2025-07-17T11:07:26.3420293Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3420507Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3420707Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3420909Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3421407Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3421735Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3422015Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3422020Z 
2025-07-17T11:07:26.3422118Z SIGABRT(6), PID: 1805793, Thread 1806088: 
2025-07-17T11:07:26.3422652Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3422932Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3423135Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3423340Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3423842Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3424054Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3424261Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3424267Z 
2025-07-17T11:07:26.3424360Z SIGABRT(6), PID: 1805795, Thread 1806282: 
2025-07-17T11:07:26.3424837Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3425041Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3425243Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3425438Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3425938Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3426145Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3426353Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3426359Z 
2025-07-17T11:07:26.3426450Z SIGABRT(6), PID: 1805793, Thread 1806091: 
2025-07-17T11:07:26.3426923Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3427124Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3427332Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3427531Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3428027Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3428235Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3428451Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3428455Z 
2025-07-17T11:07:26.3428544Z SIGABRT(6), PID: 1805794, Thread 1810690: 
2025-07-17T11:07:26.3429020Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3429224Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3429506Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3429699Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3430132Z frame #4: <unknown function> + 0x5a02023 (0x7f6c96b90023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3430757Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c96b941ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3431412Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c96b8b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3431795Z frame #7: <unknown function> + 0x79d655 (0x7f6ca8077655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3432035Z frame #8: <unknown function> + 0xdc253 (0x7f6c5f0c6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3432243Z frame #9: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3432456Z frame #10: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3432460Z 
2025-07-17T11:07:26.3432550Z SIGABRT(6), PID: 1805795, Thread 1806285: 
2025-07-17T11:07:26.3433036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3433238Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3433443Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3433639Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3434142Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3434348Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3434557Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3434561Z 
2025-07-17T11:07:26.3434648Z SIGABRT(6), PID: 1805793, Thread 1806094: 
2025-07-17T11:07:26.3435118Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3435328Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3435527Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3435727Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3436226Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3436447Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3436653Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3436657Z 
2025-07-17T11:07:26.3436748Z SIGABRT(6), PID: 1805795, Thread 1806292: 
2025-07-17T11:07:26.3437223Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3437429Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3437629Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3437903Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3438480Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3438702Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3438973Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3439033Z 
2025-07-17T11:07:26.3439127Z SIGABRT(6), PID: 1805793, Thread 1806097: 
2025-07-17T11:07:26.3439592Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3439801Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3440004Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3440201Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3440693Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3440902Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3441109Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3441115Z 
2025-07-17T11:07:26.3441205Z SIGABRT(6), PID: 1805795, Thread 1806293: 
2025-07-17T11:07:26.3441674Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3441885Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3442084Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3442289Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3442787Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3443000Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3443210Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3443214Z 
2025-07-17T11:07:26.3443310Z SIGABRT(6), PID: 1805793, Thread 1806101: 
2025-07-17T11:07:26.3443775Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3443984Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3444187Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3444392Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3444880Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3445090Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3445299Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3445310Z 
2025-07-17T11:07:26.3445397Z SIGABRT(6), PID: 1805795, Thread 1806294: 
2025-07-17T11:07:26.3445863Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3446148Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3446418Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3446618Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3447179Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3447446Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3447657Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3447661Z 
2025-07-17T11:07:26.3447750Z SIGABRT(6), PID: 1805791, Thread 1810696: 
2025-07-17T11:07:26.3448221Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3448426Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3448631Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3448827Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3449205Z frame #4: <unknown function> + 0x5a02023 (0x7fa0ed5c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3449767Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa0ed5c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3450365Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa0ed5bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3450749Z frame #7: <unknown function> + 0x79d655 (0x7fa0feaa7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3450981Z frame #8: <unknown function> + 0xdc253 (0x7fa0b5af6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3451187Z frame #9: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3451402Z frame #10: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3451406Z 
2025-07-17T11:07:26.3451496Z SIGABRT(6), PID: 1805793, Thread 1806105: 
2025-07-17T11:07:26.3451965Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3452169Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3452371Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3452569Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3453068Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3453275Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3453490Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3453495Z 
2025-07-17T11:07:26.3453581Z SIGABRT(6), PID: 1805795, Thread 1806295: 
2025-07-17T11:07:26.3454053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3454331Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3454595Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3454796Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3455355Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3455618Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3455835Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3455839Z 
2025-07-17T11:07:26.3455988Z SIGABRT(6), PID: 1805793, Thread 1806110: 
2025-07-17T11:07:26.3456454Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3456666Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3456869Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3457065Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3457554Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3457765Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3457968Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3457972Z 
2025-07-17T11:07:26.3458060Z SIGABRT(6), PID: 1805795, Thread 1806296: 
2025-07-17T11:07:26.3458524Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3458730Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3458928Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3459128Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3459622Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3459832Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3460039Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3460045Z 
2025-07-17T11:07:26.3460135Z SIGABRT(6), PID: 1805793, Thread 1806114: 
2025-07-17T11:07:26.3460595Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3460801Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3461008Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3461206Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3461701Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3461910Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3462117Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3462212Z 
2025-07-17T11:07:26.3462305Z SIGABRT(6), PID: 1805795, Thread 1806297: 
2025-07-17T11:07:26.3462841Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3463049Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3463320Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3463595Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3464088Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3464294Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3464499Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3464503Z 
2025-07-17T11:07:26.3464594Z SIGABRT(6), PID: 1805794, Thread 1810691: 
2025-07-17T11:07:26.3465059Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3465273Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3465474Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3465674Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3466048Z frame #4: <unknown function> + 0x5a02023 (0x7f6c96b90023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3466607Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c96b941ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3467200Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c96b8b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3467583Z frame #7: <unknown function> + 0x79d655 (0x7f6ca8077655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3467819Z frame #8: <unknown function> + 0xdc253 (0x7f6c5f0c6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3468022Z frame #9: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3468236Z frame #10: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3468241Z 
2025-07-17T11:07:26.3468330Z SIGABRT(6), PID: 1805793, Thread 1806118: 
2025-07-17T11:07:26.3468803Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3469009Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3469210Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3469407Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3469903Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3470109Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3470317Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3470415Z 
2025-07-17T11:07:26.3470506Z SIGABRT(6), PID: 1805795, Thread 1806298: 
2025-07-17T11:07:26.3471047Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3471254Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3471516Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3471713Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3472275Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3472481Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3472691Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3472697Z 
2025-07-17T11:07:26.3472784Z SIGABRT(6), PID: 1805795, Thread 1806299: 
2025-07-17T11:07:26.3473259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3473466Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3473669Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3473866Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3474362Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3474564Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3474775Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3474863Z SIGABRT(6), PID: 1805793, Thread 1806122: 
2025-07-17T11:07:26.3475331Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3475532Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3475744Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3475939Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3476433Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3476640Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3476848Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3476853Z 
2025-07-17T11:07:26.3476858Z 
2025-07-17T11:07:26.3476945Z SIGABRT(6), PID: 1805793, Thread 1806126: 
2025-07-17T11:07:26.3477411Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3477616Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3477816Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3478016Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3478504Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3478793Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3479062Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3479067Z 
2025-07-17T11:07:26.3479160Z SIGABRT(6), PID: 1805795, Thread 1806300: 
2025-07-17T11:07:26.3479697Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3479963Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3480165Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3480374Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3480869Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3481079Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3481284Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3481288Z 
2025-07-17T11:07:26.3481379Z SIGABRT(6), PID: 1805791, Thread 1810697: 
2025-07-17T11:07:26.3481847Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3482053Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3482251Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3482447Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3482817Z frame #4: <unknown function> + 0x5a02023 (0x7fa0ed5c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3483385Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa0ed5c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3483975Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa0ed5bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3484355Z frame #7: <unknown function> + 0x79d655 (0x7fa0feaa7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3484585Z frame #8: <unknown function> + 0xdc253 (0x7fa0b5af6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3484795Z frame #9: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3485009Z frame #10: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3485014Z 
2025-07-17T11:07:26.3485105Z SIGABRT(6), PID: 1805795, Thread 1806301: 
2025-07-17T11:07:26.3485578Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3485784Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3485988Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3486183Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3486682Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3486967Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3487243Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3487248Z 
2025-07-17T11:07:26.3487336Z SIGABRT(6), PID: 1805793, Thread 1806131: 
2025-07-17T11:07:26.3487868Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3488078Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3488346Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3488541Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3489043Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3489247Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3489459Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3489463Z 
2025-07-17T11:07:26.3489548Z SIGABRT(6), PID: 1805795, Thread 1806302: 
2025-07-17T11:07:26.3490017Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3490222Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3490429Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3490622Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3491122Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3491329Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3491535Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3491539Z 
2025-07-17T11:07:26.3491633Z SIGABRT(6), PID: 1805793, Thread 1806135: 
2025-07-17T11:07:26.3492098Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3492304Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3492507Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3492700Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3493199Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3493403Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3493609Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3493615Z 
2025-07-17T11:07:26.3493700Z SIGABRT(6), PID: 1805793, Thread 1806139: 
2025-07-17T11:07:26.3494171Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3494375Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3494583Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3494856Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3495410Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3495619Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3495888Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3495892Z 
2025-07-17T11:07:26.3496154Z SIGABRT(6), PID: 1805795, Thread 1806303: 
2025-07-17T11:07:26.3496630Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3496834Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3497035Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3497233Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3497734Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3497948Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3498153Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3498159Z 
2025-07-17T11:07:26.3498249Z SIGABRT(6), PID: 1805794, Thread 1810692: 
2025-07-17T11:07:26.3498711Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3498916Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3499119Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3499320Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3499691Z frame #4: <unknown function> + 0x5a02023 (0x7f6c96b90023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3500261Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c96b941ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3500852Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c96b8b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3501237Z frame #7: <unknown function> + 0x79d655 (0x7f6ca8077655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3501466Z frame #8: <unknown function> + 0xdc253 (0x7f6c5f0c6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3501674Z frame #9: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3501883Z frame #10: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3501887Z 
2025-07-17T11:07:26.3501981Z SIGABRT(6), PID: 1805793, Thread 1806143: 
2025-07-17T11:07:26.3502444Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3502650Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3502850Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3503149Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3503720Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3503933Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3504209Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3504215Z 
2025-07-17T11:07:26.3504309Z SIGABRT(6), PID: 1805795, Thread 1806304: 
2025-07-17T11:07:26.3504845Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3505056Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3505259Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3505457Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3505959Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3506167Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3506374Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3506384Z 
2025-07-17T11:07:26.3506471Z SIGABRT(6), PID: 1805793, Thread 1806147: 
2025-07-17T11:07:26.3506933Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3507134Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3507337Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3507531Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3508025Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3508232Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3508438Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3508443Z 
2025-07-17T11:07:26.3508530Z SIGABRT(6), PID: 1805795, Thread 1806305: 
2025-07-17T11:07:26.3508999Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3509205Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3509408Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3509607Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3510102Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3510306Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3510518Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3510522Z 
2025-07-17T11:07:26.3510609Z SIGABRT(6), PID: 1805793, Thread 1806151: 
2025-07-17T11:07:26.3511074Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3511355Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3511623Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3511821Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3512372Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3512636Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3512849Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3512853Z 
2025-07-17T11:07:26.3512942Z SIGABRT(6), PID: 1805795, Thread 1806306: 
2025-07-17T11:07:26.3513420Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3513626Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3513832Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3514023Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3514522Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3514727Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3514932Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3514936Z 
2025-07-17T11:07:26.3515026Z SIGABRT(6), PID: 1805791, Thread 1810698: 
2025-07-17T11:07:26.3515490Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3515693Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3515896Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3516093Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3516468Z frame #4: <unknown function> + 0x5a02023 (0x7fa0ed5c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3517028Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa0ed5c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3517619Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa0ed5bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3518003Z frame #7: <unknown function> + 0x79d655 (0x7fa0feaa7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3518233Z frame #8: <unknown function> + 0xdc253 (0x7fa0b5af6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3518442Z frame #9: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3518655Z frame #10: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3518659Z 
2025-07-17T11:07:26.3518748Z SIGABRT(6), PID: 1805793, Thread 1806156: 
2025-07-17T11:07:26.3519214Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3519495Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3519697Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3519961Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3520519Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3520733Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3521002Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3521009Z 
2025-07-17T11:07:26.3521101Z SIGABRT(6), PID: 1805795, Thread 1806307: 
2025-07-17T11:07:26.3521565Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3521775Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3521977Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3522172Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3522665Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3522873Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3523079Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3523083Z 
2025-07-17T11:07:26.3523174Z SIGABRT(6), PID: 1805793, Thread 1806160: 
2025-07-17T11:07:26.3523640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3523850Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3524048Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3524246Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3524738Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3524949Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3525153Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3525157Z 
2025-07-17T11:07:26.3525246Z SIGABRT(6), PID: 1805795, Thread 1806308: 
2025-07-17T11:07:26.3525708Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3525915Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3526115Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3526315Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3526809Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3527015Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3527219Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3527299Z 
2025-07-17T11:07:26.3527388Z SIGABRT(6), PID: 1805793, Thread 1806164: 
2025-07-17T11:07:26.3527918Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3528127Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3528416Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3528618Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3529171Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3529377Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3529585Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3529592Z 
2025-07-17T11:07:26.3529678Z SIGABRT(6), PID: 1805795, Thread 1806309: 
2025-07-17T11:07:26.3530150Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3530354Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3530557Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3530753Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3531244Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3531447Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3531656Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3531661Z 
2025-07-17T11:07:26.3531746Z SIGABRT(6), PID: 1805794, Thread 1810695: 
2025-07-17T11:07:26.3532215Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c60e59b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3532420Z frame #1: <unknown function> + 0x42520 (0x7f6d5270f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3532625Z frame #2: <unknown function> + 0x91117 (0x7f6d5275e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3532818Z frame #3: pthread_cond_wait + 0x211 (0x7f6d52760a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3533191Z frame #4: <unknown function> + 0x5a02023 (0x7f6c96b90023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3533746Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c96b941ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3534341Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c96b8b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3534722Z frame #7: <unknown function> + 0x79d655 (0x7f6ca8077655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3534955Z frame #8: <unknown function> + 0xdc253 (0x7f6c5f0c6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3535160Z frame #9: <unknown function> + 0x94ac3 (0x7f6d52761ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3535374Z frame #10: <unknown function> + 0x126850 (0x7f6d527f3850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3535379Z 
2025-07-17T11:07:26.3535544Z SIGABRT(6), PID: 1805793, Thread 1806168: 
2025-07-17T11:07:26.3536087Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3536374Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3536582Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3536851Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3537430Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3537641Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3537844Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3537850Z 
2025-07-17T11:07:26.3537940Z SIGABRT(6), PID: 1805795, Thread 1806310: 
2025-07-17T11:07:26.3538410Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3538618Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3538820Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3539015Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3539512Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3539721Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3539930Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3539934Z 
2025-07-17T11:07:26.3540027Z SIGABRT(6), PID: 1805793, Thread 1806172: 
2025-07-17T11:07:26.3540493Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3540698Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3540901Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3541101Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3541587Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3541793Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3541999Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3542003Z 
2025-07-17T11:07:26.3542096Z SIGABRT(6), PID: 1805795, Thread 1806311: 
2025-07-17T11:07:26.3542568Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3542773Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3542972Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3543179Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3543669Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3543970Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3544177Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3544244Z 
2025-07-17T11:07:26.3544336Z SIGABRT(6), PID: 1805793, Thread 1806176: 
2025-07-17T11:07:26.3544862Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3545070Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3545335Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3545534Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3546025Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3546237Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3546446Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3546450Z 
2025-07-17T11:07:26.3546543Z SIGABRT(6), PID: 1805795, Thread 1806312: 
2025-07-17T11:07:26.3547012Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3547220Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3547425Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3547620Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3548118Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3548324Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3548533Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3548539Z 
2025-07-17T11:07:26.3548625Z SIGABRT(6), PID: 1805791, Thread 1810699: 
2025-07-17T11:07:26.3549090Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3549294Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3549496Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3549689Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3550068Z frame #4: <unknown function> + 0x5a02023 (0x7fa0ed5c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3550633Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa0ed5c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3551227Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa0ed5bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3551608Z frame #7: <unknown function> + 0x79d655 (0x7fa0feaa7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3551844Z frame #8: <unknown function> + 0xdc253 (0x7fa0b5af6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3552126Z frame #9: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3552342Z frame #10: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3552346Z 
2025-07-17T11:07:26.3552497Z SIGABRT(6), PID: 1805793, Thread 1806180: 
2025-07-17T11:07:26.3552969Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3553237Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3554420Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3554618Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3555121Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3555331Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3555543Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3555547Z 
2025-07-17T11:07:26.3555634Z SIGABRT(6), PID: 1805795, Thread 1806313: 
2025-07-17T11:07:26.3556110Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3556313Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3556518Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3556712Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3557211Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3557417Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3557632Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3557636Z 
2025-07-17T11:07:26.3557725Z SIGABRT(6), PID: 1805793, Thread 1806184: 
2025-07-17T11:07:26.3558194Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3558399Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3558599Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3558794Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3559300Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3559512Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3559715Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3559721Z 
2025-07-17T11:07:26.3559814Z SIGABRT(6), PID: 1805795, Thread 1806314: 
2025-07-17T11:07:26.3560287Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3560494Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3560694Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3560893Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3561560Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3561778Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3561986Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3562049Z 
2025-07-17T11:07:26.3562145Z SIGABRT(6), PID: 1805793, Thread 1806188: 
2025-07-17T11:07:26.3562681Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3562885Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3563087Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3563288Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3563781Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3563988Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3564192Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3564196Z 
2025-07-17T11:07:26.3564288Z SIGABRT(6), PID: 1805795, Thread 1806315: 
2025-07-17T11:07:26.3564753Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3564956Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3565157Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3565354Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3565849Z frame #4: <unknown function> + 0x3475ab (0x7f8fe06fd5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3566064Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3566269Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3566275Z 
2025-07-17T11:07:26.3566364Z SIGABRT(6), PID: 1805793, Thread 1806193: 
2025-07-17T11:07:26.3566824Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3567029Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3567230Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3567430Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3567920Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3568131Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3568337Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3568344Z 
2025-07-17T11:07:26.3568429Z SIGABRT(6), PID: 1805795, Thread 1806850: 
2025-07-17T11:07:26.3568898Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3569429Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f8ef1f5d08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3569700Z frame #2: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3575190Z frame #3: pthread_kill + 0x12c (0x7f8fe38679fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3575529Z frame #4: raise + 0x16 (0x7f8fe3813476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3575721Z frame #5: abort + 0xd3 (0x7f8fe37f97f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3576163Z frame #6: <unknown function> + 0x972cc (0x7f8e5fae62cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3576402Z frame #7: <unknown function> + 0x95f47 (0x7f8e5fae4f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3576632Z frame #8: <unknown function> + 0x34b71 (0x7f8e5fa83b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3576855Z frame #9: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3577075Z frame #10: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3577086Z 
2025-07-17T11:07:26.3577186Z SIGABRT(6), PID: 1805793, Thread 1806198: 
2025-07-17T11:07:26.3577700Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3577919Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3578129Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3578336Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3578875Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3579098Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3579317Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3579322Z 
2025-07-17T11:07:26.3579417Z SIGABRT(6), PID: 1805795, Thread 1810582: 
2025-07-17T11:07:26.3579904Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3580123Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3580303Z frame #2: __poll + 0x4f (0x7f8fe38e9bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3580718Z frame #3: <unknown function> + 0x489f (0x7f8fe33f589f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.3580951Z frame #4: <unknown function> + 0x1483bc (0x5579e30753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3581200Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3581433Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3581667Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3581894Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3582140Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5579e305da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3582371Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3582618Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5579e305bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3582946Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3583183Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3583481Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3583719Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3584033Z frame #16: <unknown function> + 0x147734 (0x5579e3074734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3584327Z frame #17: <unknown function> + 0x21ca82 (0x5579e3149a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3584548Z frame #18: <unknown function> + 0x21c9e4 (0x5579e31499e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3584766Z frame #19: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3584981Z frame #20: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3584988Z 
2025-07-17T11:07:26.3585082Z SIGABRT(6), PID: 1805793, Thread 1806203: 
2025-07-17T11:07:26.3585599Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3585818Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3586032Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3586238Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3586754Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3586964Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3587180Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3587184Z 
2025-07-17T11:07:26.3587274Z SIGABRT(6), PID: 1805793, Thread 1806209: 
2025-07-17T11:07:26.3587755Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3587965Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3588173Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3588377Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3588886Z frame #4: <unknown function> + 0x3475ab (0x7f63081835ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3589095Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3589306Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3589311Z 
2025-07-17T11:07:26.3589402Z SIGABRT(6), PID: 1805791, Thread 1810700: 
2025-07-17T11:07:26.3589876Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3590083Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3590288Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3590487Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3590871Z frame #4: <unknown function> + 0x5a02023 (0x7fa0ed5c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3591520Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa0ed5c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3592184Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa0ed5bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3592640Z frame #7: <unknown function> + 0x79d655 (0x7fa0feaa7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3592950Z frame #8: <unknown function> + 0xdc253 (0x7fa0b5af6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3593169Z frame #9: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3593387Z frame #10: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3593394Z 
2025-07-17T11:07:26.3593491Z SIGABRT(6), PID: 1805793, Thread 1806854: 
2025-07-17T11:07:26.3593975Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3594427Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f62199e208a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3594639Z frame #2: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3594838Z frame #3: pthread_kill + 0x12c (0x7f630b2ed9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3595005Z frame #4: raise + 0x16 (0x7f630b299476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3595167Z frame #5: abort + 0xd3 (0x7f630b27f7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3595410Z frame #6: <unknown function> + 0x972cc (0x7f618756b2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3595640Z frame #7: <unknown function> + 0x95f47 (0x7f6187569f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3595865Z frame #8: <unknown function> + 0x34b71 (0x7f6187508b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3596077Z frame #9: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3596291Z frame #10: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3596296Z 
2025-07-17T11:07:26.3596391Z SIGABRT(6), PID: 1805792, Thread 1805792: 
2025-07-17T11:07:26.3596871Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3597084Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3597311Z frame #2: <unknown function> + 0x7743a (0x7f3f2148843a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3597551Z frame #3: <unknown function> + 0x76c8e (0x7f3f21487c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3597779Z frame #4: <unknown function> + 0x6ad69 (0x7f3f2147bd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3598005Z frame #5: <unknown function> + 0x3f54db (0x7f3fb20e04db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3598223Z frame #6: <unknown function> + 0x3e11d8 (0x7f3fb20cc1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3598440Z frame #7: <unknown function> + 0xb2f86 (0x7f3fb1d9df86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3598648Z frame #8: <unknown function> + 0x8e501 (0x7f3fb1d79501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3599044Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f3fdcb807ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.3599433Z frame #10: <unknown function> + 0xd307d9 (0x7f3ffb0d37d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3599768Z frame #11: <unknown function> + 0x139902 (0x5585f1e90902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3600083Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5585f1e87a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3600320Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3600623Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5585f1e87a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3600914Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3601155Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3601383Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3601621Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3601845Z frame #19: <unknown function> + 0x1474b2 (0x5585f1e9e4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3602052Z frame #20: PyObject_Call + 0xbc (0x5585f1e9ee4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3602276Z frame #21: <unknown function> + 0x1ff69d (0x5585f1f5669d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3602502Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5585f1e8c023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3602752Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5585f1e87cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3602977Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3603216Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5585f1e833bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3603439Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3603678Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5585f1e833bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3603898Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3604142Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5585f1e87a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3604401Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5585f1e8b4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3604625Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5585f1e9cab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3604843Z frame #32: <unknown function> + 0x205949 (0x5585f1f5c949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3605071Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5585f1e8c023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3605312Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5585f1e87cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3605538Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3605775Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5585f1e833bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3606001Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3606238Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5585f1e833bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3606464Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3606701Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5585f1e833bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3606926Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3607165Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5585f1e843f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3607464Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3607917Z frame #44: <unknown function> + 0x7ab3f7 (0x7f3ffab4e3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3608145Z frame #45: <unknown function> + 0x13bc88 (0x5585f1e92c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3608412Z frame #46: PyObject_Call + 0x20f (0x5585f1e9ef9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3608663Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5585f1e8898d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3608944Z frame #48: <unknown function> + 0x147608 (0x5585f1e9e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3609190Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5585f1e85c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3609413Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3609653Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5585f1e833bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3609877Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3610118Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5585f1e843f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3610343Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3610580Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5585f1e833bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3610802Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3611044Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5585f1e85c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3611268Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3611513Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5585f1e85c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3611733Z frame #60: <unknown function> + 0x147608 (0x5585f1e9e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3611974Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5585f1e85c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3612192Z frame #62: <unknown function> + 0x147608 (0x5585f1e9e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3612197Z 
2025-07-17T11:07:26.3612296Z SIGABRT(6), PID: 1805795, Thread 1810583: 
2025-07-17T11:07:26.3612778Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3612996Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3613206Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3613437Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f8fe38652dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3613883Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f8f1dca5048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3614315Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f8f1dca7bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3614547Z frame #6: <unknown function> + 0xdc253 (0x7f8ef01c9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3614761Z frame #7: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3614977Z frame #8: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3614981Z 
2025-07-17T11:07:26.3615075Z SIGABRT(6), PID: 1805793, Thread 1810178: 
2025-07-17T11:07:26.3615628Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3615976Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3616155Z frame #2: __poll + 0x4f (0x7f630b36fbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3616647Z frame #3: <unknown function> + 0x489f (0x7f630ae7b89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.3616951Z frame #4: <unknown function> + 0x1483bc (0x556ed0d3d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3617197Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3617425Z frame #6: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3617666Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3617890Z frame #8: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3618137Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x556ed0d25a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3618368Z frame #10: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3618619Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x556ed0d23c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3618849Z frame #12: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3619094Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3619318Z frame #14: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3619556Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3619784Z frame #16: <unknown function> + 0x147734 (0x556ed0d3c734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3620011Z frame #17: <unknown function> + 0x21ca82 (0x556ed0e11a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3620228Z frame #18: <unknown function> + 0x21c9e4 (0x556ed0e119e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3620452Z frame #19: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3620666Z frame #20: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3620673Z 
2025-07-17T11:07:26.3620771Z SIGABRT(6), PID: 1805792, Thread 1805965: 
2025-07-17T11:07:26.3621260Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3621479Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3621684Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3621891Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3622416Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3622638Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3622855Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3622861Z 
2025-07-17T11:07:26.3622953Z SIGABRT(6), PID: 1805792, Thread 1805970: 
2025-07-17T11:07:26.3623443Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3623743Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3624022Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3624228Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3624798Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3625091Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3625306Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3625310Z 
2025-07-17T11:07:26.3625401Z SIGABRT(6), PID: 1805795, Thread 1810584: 
2025-07-17T11:07:26.3625883Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3626095Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3626303Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3626527Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f8fe38652dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3627010Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f8f1dc9f741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3627247Z frame #5: <unknown function> + 0xdc253 (0x7f8ef01c9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3627456Z frame #6: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3627665Z frame #7: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3627671Z 
2025-07-17T11:07:26.3627765Z SIGABRT(6), PID: 1805795, Thread 1810587: 
2025-07-17T11:07:26.3628237Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3628446Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3628617Z frame #2: ioctl + 0x3f (0x7f8fe38eb94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3628858Z frame #3: <unknown function> + 0x13eef0 (0x7f8e5fb8def0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3629090Z frame #4: <unknown function> + 0x1378c4 (0x7f8e5fb868c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3629324Z frame #5: <unknown function> + 0x95b2b (0x7f8e5fae4b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3629549Z frame #6: <unknown function> + 0x34b71 (0x7f8e5fa83b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3629760Z frame #7: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3629970Z frame #8: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3629975Z 
2025-07-17T11:07:26.3630067Z SIGABRT(6), PID: 1805792, Thread 1805974: 
2025-07-17T11:07:26.3630536Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3630746Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3630948Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3631149Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3631653Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3631941Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3632208Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3632214Z 
2025-07-17T11:07:26.3632310Z SIGABRT(6), PID: 1805793, Thread 1810179: 
2025-07-17T11:07:26.3632841Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3633108Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3633315Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3633543Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f630b2eb2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3633988Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f624572a048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3634411Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f624572cbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3634639Z frame #6: <unknown function> + 0xdc253 (0x7f6217c4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3634853Z frame #7: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3635063Z frame #8: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3635070Z 
2025-07-17T11:07:26.3635161Z SIGABRT(6), PID: 1805792, Thread 1805979: 
2025-07-17T11:07:26.3635641Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3635850Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3636054Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3636256Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3636760Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3636968Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3637181Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3637185Z 
2025-07-17T11:07:26.3637274Z SIGABRT(6), PID: 1805795, Thread 1810599: 
2025-07-17T11:07:26.3637750Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3637958Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3638129Z frame #2: __poll + 0x4f (0x7f8fe38e9bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3638345Z frame #3: <unknown function> + 0xd4540 (0x7f8ea7cfd540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3638554Z frame #4: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3638764Z frame #5: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3638770Z 
2025-07-17T11:07:26.3638860Z SIGABRT(6), PID: 1805791, Thread 1810701: 
2025-07-17T11:07:26.3639329Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa0b7889b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3639536Z frame #1: <unknown function> + 0x42520 (0x7fa1a9142520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3639813Z frame #2: <unknown function> + 0x91117 (0x7fa1a9191117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3640075Z frame #3: pthread_cond_wait + 0x211 (0x7fa1a9193a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3640455Z frame #4: <unknown function> + 0x5a02023 (0x7fa0ed5c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3641097Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa0ed5c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3641757Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa0ed5bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3642148Z frame #7: <unknown function> + 0x79d655 (0x7fa0feaa7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3642381Z frame #8: <unknown function> + 0xdc253 (0x7fa0b5af6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3642593Z frame #9: <unknown function> + 0x94ac3 (0x7fa1a9194ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3642808Z frame #10: <unknown function> + 0x126850 (0x7fa1a9226850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3642813Z 
2025-07-17T11:07:26.3642906Z SIGABRT(6), PID: 1805795, Thread 1810606: 
2025-07-17T11:07:26.3643379Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3643597Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3643764Z frame #2: __poll + 0x4f (0x7f8fe38e9bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3643983Z frame #3: <unknown function> + 0xd65a2 (0x7f8ea7cff5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3644191Z frame #4: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3644405Z frame #5: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3644409Z 
2025-07-17T11:07:26.3644498Z SIGABRT(6), PID: 1805792, Thread 1805983: 
2025-07-17T11:07:26.3644975Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3645181Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3645386Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3645583Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3646086Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3646297Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3646506Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3646510Z 
2025-07-17T11:07:26.3646602Z SIGABRT(6), PID: 1805795, Thread 1810620: 
2025-07-17T11:07:26.3647077Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3647288Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3647494Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3647692Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3647987Z frame #4: <unknown function> + 0xd1e7f (0x7f8ea7cfae7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3648257Z frame #5: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3648470Z frame #6: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3648474Z 
2025-07-17T11:07:26.3648566Z SIGABRT(6), PID: 1805793, Thread 1810180: 
2025-07-17T11:07:26.3649100Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3649366Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3649570Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3649794Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f630b2eb2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3650274Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f6245724741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3650509Z frame #5: <unknown function> + 0xdc253 (0x7f6217c4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3650715Z frame #6: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3650924Z frame #7: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3650928Z 
2025-07-17T11:07:26.3651018Z SIGABRT(6), PID: 1805792, Thread 1805988: 
2025-07-17T11:07:26.3651486Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3651693Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3651897Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3652092Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3652596Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3652804Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3653017Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3653022Z 
2025-07-17T11:07:26.3653109Z SIGABRT(6), PID: 1805793, Thread 1810181: 
2025-07-17T11:07:26.3653576Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3653781Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3653950Z frame #2: ioctl + 0x3f (0x7f630b37194f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3654185Z frame #3: <unknown function> + 0x13eef0 (0x7f6187612ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3654415Z frame #4: <unknown function> + 0x1378c4 (0x7f618760b8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3654646Z frame #5: <unknown function> + 0x95b2b (0x7f6187569b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3654873Z frame #6: <unknown function> + 0x34b71 (0x7f6187508b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3655082Z frame #7: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3655293Z frame #8: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3655297Z 
2025-07-17T11:07:26.3655385Z SIGABRT(6), PID: 1805795, Thread 1810627: 
2025-07-17T11:07:26.3655997Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3656284Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3656497Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3656799Z frame #3: <unknown function> + 0x9ca53 (0x7f8fe386da53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3657064Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5579e3041bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3657365Z frame #5: <unknown function> + 0x1d314f (0x5579e310014f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3657591Z frame #6: <unknown function> + 0x1d2f6b (0x5579e30fff6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3657813Z frame #7: <unknown function> + 0x6df36 (0x5579e2f9af36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3658064Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3658290Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3658533Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3658760Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3659000Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3659225Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3659460Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3659685Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3659923Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3660138Z frame #17: <unknown function> + 0x147734 (0x5579e3074734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3660363Z frame #18: <unknown function> + 0x21ca82 (0x5579e3149a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3660582Z frame #19: <unknown function> + 0x21c9e4 (0x5579e31499e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3660797Z frame #20: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3661013Z frame #21: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3661018Z 
2025-07-17T11:07:26.3661111Z SIGABRT(6), PID: 1805792, Thread 1805993: 
2025-07-17T11:07:26.3661584Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3661800Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3662006Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3662211Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3662717Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3662926Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3663137Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3663143Z 
2025-07-17T11:07:26.3663232Z SIGABRT(6), PID: 1805793, Thread 1810600: 
2025-07-17T11:07:26.3663708Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3664006Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3664254Z frame #2: __poll + 0x4f (0x7f630b36fbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3664472Z frame #3: <unknown function> + 0xd4540 (0x7f61cf782540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3664744Z frame #4: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3664955Z frame #5: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3665020Z 
2025-07-17T11:07:26.3665115Z SIGABRT(6), PID: 1805792, Thread 1805998: 
2025-07-17T11:07:26.3665585Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3665792Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3665997Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3666198Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3666701Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3666908Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3667116Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3667121Z 
2025-07-17T11:07:26.3667214Z SIGABRT(6), PID: 1805795, Thread 1810640: 
2025-07-17T11:07:26.3667683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3667891Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3668055Z frame #2: read + 0x4c (0x7f8fe38e581c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3668241Z frame #3: _Py_read + 0x4c (0x5579e3107a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3668464Z frame #4: <unknown function> + 0x208c3b (0x5579e3135c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3668689Z frame #5: <unknown function> + 0x14648d (0x5579e307348d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3668935Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5579e3082215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3669156Z frame #7: <unknown function> + 0x208ae7 (0x5579e3135ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3669374Z frame #8: <unknown function> + 0x208472 (0x5579e3135472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3669596Z frame #9: <unknown function> + 0x1da6ee (0x5579e31076ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3669820Z frame #10: <unknown function> + 0x1483bc (0x5579e30753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3670066Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3670291Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3670530Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5579e30593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3670745Z frame #14: <unknown function> + 0x147734 (0x5579e3074734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3670999Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5579e305bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3671223Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3671463Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3671764Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5579e30690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3672067Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5579e30597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3672286Z frame #20: <unknown function> + 0x147734 (0x5579e3074734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3672568Z frame #21: <unknown function> + 0x21ca82 (0x5579e3149a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3672789Z frame #22: <unknown function> + 0x21c9e4 (0x5579e31499e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3673068Z frame #23: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3673281Z frame #24: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3673286Z 
2025-07-17T11:07:26.3673379Z SIGABRT(6), PID: 1805793, Thread 1810607: 
2025-07-17T11:07:26.3673855Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3674069Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3674235Z frame #2: __poll + 0x4f (0x7f630b36fbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3674454Z frame #3: <unknown function> + 0xd65a2 (0x7f61cf7845a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3674661Z frame #4: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3674872Z frame #5: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3674877Z 
2025-07-17T11:07:26.3674965Z SIGABRT(6), PID: 1805792, Thread 1806003: 
2025-07-17T11:07:26.3675443Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3675650Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3675857Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3676057Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3676563Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3676769Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3676978Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3676982Z 
2025-07-17T11:07:26.3677071Z SIGABRT(6), PID: 1805793, Thread 1810619: 
2025-07-17T11:07:26.3677542Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3677752Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3677957Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3678156Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3678373Z frame #4: <unknown function> + 0xd1e7f (0x7f61cf77fe7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3678581Z frame #5: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3678792Z frame #6: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3678801Z 
2025-07-17T11:07:26.3678891Z SIGABRT(6), PID: 1805792, Thread 1806008: 
2025-07-17T11:07:26.3679362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3679705Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3679911Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3680110Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3680674Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3680938Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3681153Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3681157Z 
2025-07-17T11:07:26.3681245Z SIGABRT(6), PID: 1805793, Thread 1810623: 
2025-07-17T11:07:26.3681720Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3681930Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3682136Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3682351Z frame #3: <unknown function> + 0x9ca53 (0x7f630b2f3a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3682606Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x556ed0d09bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3682837Z frame #5: <unknown function> + 0x1d314f (0x556ed0dc814f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3683061Z frame #6: <unknown function> + 0x1d2f6b (0x556ed0dc7f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3683283Z frame #7: <unknown function> + 0x6df36 (0x556ed0c62f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3683528Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3683759Z frame #9: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3683998Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3684230Z frame #11: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3684464Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3684692Z frame #13: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3684928Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3685156Z frame #15: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3685392Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3685617Z frame #17: <unknown function> + 0x147734 (0x556ed0d3c734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3685836Z frame #18: <unknown function> + 0x21ca82 (0x556ed0e11a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3686058Z frame #19: <unknown function> + 0x21c9e4 (0x556ed0e119e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3686269Z frame #20: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3686482Z frame #21: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3686493Z 
2025-07-17T11:07:26.3686584Z SIGABRT(6), PID: 1805792, Thread 1806014: 
2025-07-17T11:07:26.3687061Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3687345Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3687634Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3687836Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3688398Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3688665Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3688878Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3688883Z 
2025-07-17T11:07:26.3688972Z SIGABRT(6), PID: 1805793, Thread 1810635: 
2025-07-17T11:07:26.3689445Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3689653Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3689826Z frame #2: read + 0x4c (0x7f630b36b81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3690008Z frame #3: _Py_read + 0x4c (0x556ed0dcfa6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3690237Z frame #4: <unknown function> + 0x208c3b (0x556ed0dfdc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3690459Z frame #5: <unknown function> + 0x14648d (0x556ed0d3b48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3690706Z frame #6: PyObject_VectorcallMethod + 0x85 (0x556ed0d4a215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3690927Z frame #7: <unknown function> + 0x208ae7 (0x556ed0dfdae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3691148Z frame #8: <unknown function> + 0x208472 (0x556ed0dfd472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3691371Z frame #9: <unknown function> + 0x1da6ee (0x556ed0dcf6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3691599Z frame #10: <unknown function> + 0x1483bc (0x556ed0d3d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3691841Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3692071Z frame #12: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3692307Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x556ed0d213bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3692527Z frame #14: <unknown function> + 0x147734 (0x556ed0d3c734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3692774Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x556ed0d23c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3693003Z frame #16: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3693242Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3693478Z frame #18: _PyFunction_Vectorcall + 0x6c (0x556ed0d310ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3693713Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x556ed0d217b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3693935Z frame #20: <unknown function> + 0x147734 (0x556ed0d3c734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3694157Z frame #21: <unknown function> + 0x21ca82 (0x556ed0e11a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3694378Z frame #22: <unknown function> + 0x21c9e4 (0x556ed0e119e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3694589Z frame #23: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3694801Z frame #24: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3694875Z 
2025-07-17T11:07:26.3694969Z SIGABRT(6), PID: 1805792, Thread 1806019: 
2025-07-17T11:07:26.3695510Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3695726Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3696090Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3696314Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3696926Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3697142Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3697361Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3697366Z 
2025-07-17T11:07:26.3697456Z SIGABRT(6), PID: 1805792, Thread 1806024: 
2025-07-17T11:07:26.3697944Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3698152Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3698356Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3698557Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3699068Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3699283Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3699502Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3699505Z 
2025-07-17T11:07:26.3699595Z SIGABRT(6), PID: 1805792, Thread 1806030: 
2025-07-17T11:07:26.3700080Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3700291Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3700500Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3700699Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3701207Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3701414Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3701628Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3701636Z 
2025-07-17T11:07:26.3701724Z SIGABRT(6), PID: 1805795, Thread 1810678: 
2025-07-17T11:07:26.3702202Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3702411Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3702617Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3702819Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3703200Z frame #4: <unknown function> + 0x5a02023 (0x7f8f27c93023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3703937Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8f27c971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3704606Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8f27c8e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3704997Z frame #7: <unknown function> + 0x79d655 (0x7f8f3917e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3705295Z frame #8: <unknown function> + 0xdc253 (0x7f8ef01c9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3705510Z frame #9: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3705726Z frame #10: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3705733Z 
2025-07-17T11:07:26.3705825Z SIGABRT(6), PID: 1805792, Thread 1806035: 
2025-07-17T11:07:26.3706303Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3706522Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3706731Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3706933Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3707437Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3707646Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3707854Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3707859Z 
2025-07-17T11:07:26.3707953Z SIGABRT(6), PID: 1805792, Thread 1806039: 
2025-07-17T11:07:26.3708419Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3708628Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3708833Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3709033Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3709532Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3709741Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3709949Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3709953Z 
2025-07-17T11:07:26.3710048Z SIGABRT(6), PID: 1805792, Thread 1806043: 
2025-07-17T11:07:26.3710517Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3710725Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3710931Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3711131Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3711630Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3711917Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3712189Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3712197Z 
2025-07-17T11:07:26.3712287Z SIGABRT(6), PID: 1805793, Thread 1810662: 
2025-07-17T11:07:26.3712823Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3713031Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3713296Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3713497Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3713872Z frame #4: <unknown function> + 0x5a02023 (0x7f624f718023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3714441Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f624f71c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3715037Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f624f7131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3715418Z frame #7: <unknown function> + 0x79d655 (0x7f6260c03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3715652Z frame #8: <unknown function> + 0xdc253 (0x7f6217c4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3715860Z frame #9: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3716077Z frame #10: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3716083Z 
2025-07-17T11:07:26.3716173Z SIGABRT(6), PID: 1805792, Thread 1806046: 
2025-07-17T11:07:26.3716651Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3716859Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3717066Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3717264Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3717767Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3717974Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3718187Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3718191Z 
2025-07-17T11:07:26.3718280Z SIGABRT(6), PID: 1805792, Thread 1806050: 
2025-07-17T11:07:26.3718762Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3718968Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3719176Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3719374Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3719879Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3720184Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3720399Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3720462Z 
2025-07-17T11:07:26.3720556Z SIGABRT(6), PID: 1805792, Thread 1806054: 
2025-07-17T11:07:26.3721090Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3721299Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3721567Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3721770Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3722282Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3722489Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3722703Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3722708Z 
2025-07-17T11:07:26.3722796Z SIGABRT(6), PID: 1805795, Thread 1810679: 
2025-07-17T11:07:26.3723274Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3723484Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3723689Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3723888Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3724262Z frame #4: <unknown function> + 0x5a02023 (0x7f8f27c93023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3724833Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8f27c971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3725426Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8f27c8e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3725817Z frame #7: <unknown function> + 0x79d655 (0x7f8f3917e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3726050Z frame #8: <unknown function> + 0xdc253 (0x7f8ef01c9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3726258Z frame #9: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3726474Z frame #10: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3726478Z 
2025-07-17T11:07:26.3726572Z SIGABRT(6), PID: 1805792, Thread 1806058: 
2025-07-17T11:07:26.3727048Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3727260Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3727468Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3727668Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3728167Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3728374Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3728659Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3728663Z 
2025-07-17T11:07:26.3728816Z SIGABRT(6), PID: 1805792, Thread 1806062: 
2025-07-17T11:07:26.3729292Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3729561Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3729827Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3730031Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3730528Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3730736Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3730945Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3730950Z 
2025-07-17T11:07:26.3731045Z SIGABRT(6), PID: 1805793, Thread 1810663: 
2025-07-17T11:07:26.3731518Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3731727Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3731931Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3732129Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3732503Z frame #4: <unknown function> + 0x5a02023 (0x7f624f718023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3733071Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f624f71c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3733668Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f624f7131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3734048Z frame #7: <unknown function> + 0x79d655 (0x7f6260c03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3734279Z frame #8: <unknown function> + 0xdc253 (0x7f6217c4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3734485Z frame #9: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3734701Z frame #10: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3734707Z 
2025-07-17T11:07:26.3734799Z SIGABRT(6), PID: 1805792, Thread 1806066: 
2025-07-17T11:07:26.3735276Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3735484Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3735692Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3735890Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3736451Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3736659Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3736965Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3736970Z 
2025-07-17T11:07:26.3737061Z SIGABRT(6), PID: 1805792, Thread 1806070: 
2025-07-17T11:07:26.3737610Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3737889Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3738097Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3738365Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3738866Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3739073Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3739285Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3739289Z 
2025-07-17T11:07:26.3739380Z SIGABRT(6), PID: 1805792, Thread 1806073: 
2025-07-17T11:07:26.3739853Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3740059Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3740265Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3740459Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3740959Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3741170Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3741382Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3741386Z 
2025-07-17T11:07:26.3741475Z SIGABRT(6), PID: 1805795, Thread 1810680: 
2025-07-17T11:07:26.3741947Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3742151Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3742358Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3742557Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3742930Z frame #4: <unknown function> + 0x5a02023 (0x7f8f27c93023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3743507Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8f27c971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3744100Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8f27c8e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3744484Z frame #7: <unknown function> + 0x79d655 (0x7f8f3917e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3744715Z frame #8: <unknown function> + 0xdc253 (0x7f8ef01c9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3744923Z frame #9: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3745136Z frame #10: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3745320Z SIGABRT(6), PID: 1805792, Thread 1806076: 
2025-07-17T11:07:26.3745855Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3746069Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3746329Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3746534Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3747111Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3747322Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3747536Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3747540Z 
2025-07-17T11:07:26.3747544Z 
2025-07-17T11:07:26.3747635Z SIGABRT(6), PID: 1805792, Thread 1806080: 
2025-07-17T11:07:26.3748113Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3748322Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3748524Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3748723Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3749219Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3749431Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3749640Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3749735Z SIGABRT(6), PID: 1805793, Thread 1810664: 
2025-07-17T11:07:26.3750203Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3750413Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3750617Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3750819Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3751189Z frame #4: <unknown function> + 0x5a02023 (0x7f624f718023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3751760Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f624f71c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3752353Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f624f7131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3752739Z frame #7: <unknown function> + 0x79d655 (0x7f6260c03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3752971Z frame #8: <unknown function> + 0xdc253 (0x7f6217c4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3753183Z frame #9: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3753394Z frame #10: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3753473Z 
2025-07-17T11:07:26.3753476Z 
2025-07-17T11:07:26.3753568Z SIGABRT(6), PID: 1805792, Thread 1806083: 
2025-07-17T11:07:26.3754103Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3754314Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3754574Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3754774Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3755351Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3755557Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3755771Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3755775Z 
2025-07-17T11:07:26.3755869Z SIGABRT(6), PID: 1805795, Thread 1810681: 
2025-07-17T11:07:26.3756342Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3756549Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3756755Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3756956Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3757328Z frame #4: <unknown function> + 0x5a02023 (0x7f8f27c93023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3757891Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8f27c971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3758493Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8f27c8e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3758877Z frame #7: <unknown function> + 0x79d655 (0x7f8f3917e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3759109Z frame #8: <unknown function> + 0xdc253 (0x7f8ef01c9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3759318Z frame #9: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3759539Z frame #10: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3759543Z 
2025-07-17T11:07:26.3759631Z SIGABRT(6), PID: 1805792, Thread 1806086: 
2025-07-17T11:07:26.3760109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3760321Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3760526Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3760724Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3761225Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3761433Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3761645Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3761719Z 
2025-07-17T11:07:26.3761814Z SIGABRT(6), PID: 1805792, Thread 1806089: 
2025-07-17T11:07:26.3762347Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3762557Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3762759Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3763017Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3763569Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3763778Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3763990Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3763996Z 
2025-07-17T11:07:26.3764090Z SIGABRT(6), PID: 1805792, Thread 1806092: 
2025-07-17T11:07:26.3764560Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3764765Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3764970Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3765169Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3765664Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3765876Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3766086Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3766090Z 
2025-07-17T11:07:26.3766183Z SIGABRT(6), PID: 1805792, Thread 1806095: 
2025-07-17T11:07:26.3766650Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3766858Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3767062Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3767263Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3767757Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3767967Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3768177Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3768181Z 
2025-07-17T11:07:26.3768275Z SIGABRT(6), PID: 1805793, Thread 1810665: 
2025-07-17T11:07:26.3768744Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3768954Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3769158Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3769360Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3769734Z frame #4: <unknown function> + 0x5a02023 (0x7f624f718023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3770375Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f624f71c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3771031Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f624f7131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3771480Z frame #7: <unknown function> + 0x79d655 (0x7f6260c03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3771772Z frame #8: <unknown function> + 0xdc253 (0x7f6217c4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3771988Z frame #9: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3772201Z frame #10: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3772210Z 
2025-07-17T11:07:26.3772302Z SIGABRT(6), PID: 1805792, Thread 1806098: 
2025-07-17T11:07:26.3772779Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3772984Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3773191Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3773388Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3773885Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3774093Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3774308Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3774314Z 
2025-07-17T11:07:26.3774404Z SIGABRT(6), PID: 1805792, Thread 1806102: 
2025-07-17T11:07:26.3774883Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3775089Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3775295Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3775493Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3776054Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3776263Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3776476Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3776480Z 
2025-07-17T11:07:26.3776570Z SIGABRT(6), PID: 1805795, Thread 1810682: 
2025-07-17T11:07:26.3777050Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3777256Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3777467Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3777665Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3778040Z frame #4: <unknown function> + 0x5a02023 (0x7f8f27c93023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3778603Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8f27c971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3779363Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8f27c8e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3779839Z frame #7: <unknown function> + 0x79d655 (0x7f8f3917e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3780153Z frame #8: <unknown function> + 0xdc253 (0x7f8ef01c9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3780363Z frame #9: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3780582Z frame #10: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3780586Z 
2025-07-17T11:07:26.3780677Z SIGABRT(6), PID: 1805792, Thread 1806107: 
2025-07-17T11:07:26.3781151Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3781359Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3781564Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3781762Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3782266Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3782471Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3782682Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3782688Z 
2025-07-17T11:07:26.3782779Z SIGABRT(6), PID: 1805792, Thread 1806111: 
2025-07-17T11:07:26.3783250Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3783458Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3783662Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3783863Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3784361Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3784573Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3784784Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3784789Z 
2025-07-17T11:07:26.3784880Z SIGABRT(6), PID: 1805792, Thread 1806115: 
2025-07-17T11:07:26.3785346Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3785554Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3785759Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3785960Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3786453Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3786662Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3786947Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3786951Z 
2025-07-17T11:07:26.3787103Z SIGABRT(6), PID: 1805792, Thread 1806119: 
2025-07-17T11:07:26.3787580Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3787849Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3788114Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3788316Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3788815Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3789026Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3789234Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3789240Z 
2025-07-17T11:07:26.3789331Z SIGABRT(6), PID: 1805793, Thread 1810666: 
2025-07-17T11:07:26.3789801Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3790008Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3790210Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3790408Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3790778Z frame #4: <unknown function> + 0x5a02023 (0x7f624f718023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3791348Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f624f71c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3791939Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f624f7131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3792326Z frame #7: <unknown function> + 0x79d655 (0x7f6260c03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3792560Z frame #8: <unknown function> + 0xdc253 (0x7f6217c4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3792768Z frame #9: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3792983Z frame #10: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3792988Z 
2025-07-17T11:07:26.3793079Z SIGABRT(6), PID: 1805792, Thread 1806123: 
2025-07-17T11:07:26.3793553Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3793760Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3793967Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3794163Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3794666Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3794875Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3795163Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3795167Z 
2025-07-17T11:07:26.3795256Z SIGABRT(6), PID: 1805792, Thread 1806127: 
2025-07-17T11:07:26.3795793Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3796058Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3796268Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3796528Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3797034Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3797240Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3797454Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3797458Z 
2025-07-17T11:07:26.3797547Z SIGABRT(6), PID: 1805795, Thread 1810683: 
2025-07-17T11:07:26.3798020Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3798229Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3798441Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3798638Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3799014Z frame #4: <unknown function> + 0x5a02023 (0x7f8f27c93023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3799580Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8f27c971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3800173Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8f27c8e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3800556Z frame #7: <unknown function> + 0x79d655 (0x7f8f3917e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3800791Z frame #8: <unknown function> + 0xdc253 (0x7f8ef01c9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3800998Z frame #9: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3801212Z frame #10: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3801218Z 
2025-07-17T11:07:26.3801308Z SIGABRT(6), PID: 1805792, Thread 1806130: 
2025-07-17T11:07:26.3801784Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3801992Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3802195Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3802389Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3802891Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3803102Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3803311Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3803384Z 
2025-07-17T11:07:26.3803478Z SIGABRT(6), PID: 1805792, Thread 1806134: 
2025-07-17T11:07:26.3804011Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3804222Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3804483Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3804744Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3805249Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3805455Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3805666Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3805670Z 
2025-07-17T11:07:26.3805764Z SIGABRT(6), PID: 1805792, Thread 1806138: 
2025-07-17T11:07:26.3806236Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3806448Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3806650Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3806857Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3807350Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3807561Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3807770Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3807774Z 
2025-07-17T11:07:26.3807867Z SIGABRT(6), PID: 1805793, Thread 1810667: 
2025-07-17T11:07:26.3808336Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3808543Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3808746Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3808945Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3809316Z frame #4: <unknown function> + 0x5a02023 (0x7f624f718023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3809890Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f624f71c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3810481Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f624f7131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3810862Z frame #7: <unknown function> + 0x79d655 (0x7f6260c03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3811091Z frame #8: <unknown function> + 0xdc253 (0x7f6217c4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3811300Z frame #9: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3811514Z frame #10: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3811599Z 
2025-07-17T11:07:26.3811692Z SIGABRT(6), PID: 1805792, Thread 1806142: 
2025-07-17T11:07:26.3812249Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3812460Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3812724Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3812925Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3813487Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3813695Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3813907Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3813911Z 
2025-07-17T11:07:26.3814000Z SIGABRT(6), PID: 1805792, Thread 1806146: 
2025-07-17T11:07:26.3814473Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3814676Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3814884Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3815080Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3815579Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3815784Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3816053Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3816057Z 
2025-07-17T11:07:26.3816148Z SIGABRT(6), PID: 1805795, Thread 1810684: 
2025-07-17T11:07:26.3816623Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3816829Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3817035Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3817235Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3817608Z frame #4: <unknown function> + 0x5a02023 (0x7f8f27c93023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3818171Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8f27c971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3818766Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8f27c8e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3819145Z frame #7: <unknown function> + 0x79d655 (0x7f8f3917e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3819379Z frame #8: <unknown function> + 0xdc253 (0x7f8ef01c9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3819586Z frame #9: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3819802Z frame #10: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3819897Z 
2025-07-17T11:07:26.3819991Z SIGABRT(6), PID: 1805792, Thread 1806150: 
2025-07-17T11:07:26.3820539Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3820751Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3820959Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3821229Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3821829Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3822037Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3822252Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3822258Z 
2025-07-17T11:07:26.3822347Z SIGABRT(6), PID: 1805792, Thread 1806154: 
2025-07-17T11:07:26.3822821Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3823031Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3823237Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3823436Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3823940Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3824151Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3824364Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3824368Z 
2025-07-17T11:07:26.3824460Z SIGABRT(6), PID: 1805792, Thread 1806158: 
2025-07-17T11:07:26.3824934Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3825144Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3825345Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3825545Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3826042Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3826252Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3826463Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3826467Z 
2025-07-17T11:07:26.3826562Z SIGABRT(6), PID: 1805793, Thread 1810668: 
2025-07-17T11:07:26.3827030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3827239Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3827444Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3827646Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3828016Z frame #4: <unknown function> + 0x5a02023 (0x7f624f718023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3828664Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f624f71c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3829321Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f624f7131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3829767Z frame #7: <unknown function> + 0x79d655 (0x7f6260c03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3830063Z frame #8: <unknown function> + 0xdc253 (0x7f6217c4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3830282Z frame #9: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3830498Z frame #10: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3830506Z 
2025-07-17T11:07:26.3830602Z SIGABRT(6), PID: 1805792, Thread 1806163: 
2025-07-17T11:07:26.3831084Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3831296Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3831504Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3831711Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3832222Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3832434Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3832646Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3832655Z 
2025-07-17T11:07:26.3832745Z SIGABRT(6), PID: 1805792, Thread 1806167: 
2025-07-17T11:07:26.3833224Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3833431Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3833646Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3833845Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3834346Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3834555Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3834766Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3834771Z 
2025-07-17T11:07:26.3834859Z SIGABRT(6), PID: 1805795, Thread 1810685: 
2025-07-17T11:07:26.3835330Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8ef1f5cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3835535Z frame #1: <unknown function> + 0x42520 (0x7f8fe3813520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3835742Z frame #2: <unknown function> + 0x91117 (0x7f8fe3862117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3835944Z frame #3: pthread_cond_wait + 0x211 (0x7f8fe3864a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3836320Z frame #4: <unknown function> + 0x5a02023 (0x7f8f27c93023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3836884Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8f27c971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3837609Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8f27c8e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3838048Z frame #7: <unknown function> + 0x79d655 (0x7f8f3917e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3838340Z frame #8: <unknown function> + 0xdc253 (0x7f8ef01c9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3838550Z frame #9: <unknown function> + 0x94ac3 (0x7f8fe3865ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3838765Z frame #10: <unknown function> + 0x126850 (0x7f8fe38f7850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3838769Z 
2025-07-17T11:07:26.3838866Z SIGABRT(6), PID: 1805792, Thread 1806171: 
2025-07-17T11:07:26.3839340Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3839547Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3839751Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3839951Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3840452Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3840659Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3840872Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3840878Z 
2025-07-17T11:07:26.3840967Z SIGABRT(6), PID: 1805792, Thread 1806175: 
2025-07-17T11:07:26.3841443Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3841652Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3841857Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3842054Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3842551Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3842760Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3842971Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3842976Z 
2025-07-17T11:07:26.3843066Z SIGABRT(6), PID: 1805793, Thread 1810669: 
2025-07-17T11:07:26.3843531Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f62199e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3843739Z frame #1: <unknown function> + 0x42520 (0x7f630b299520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3843950Z frame #2: <unknown function> + 0x91117 (0x7f630b2e8117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3844153Z frame #3: pthread_cond_wait + 0x211 (0x7f630b2eaa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3844524Z frame #4: <unknown function> + 0x5a02023 (0x7f624f718023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3845088Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f624f71c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3845842Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f624f7131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3846286Z frame #7: <unknown function> + 0x79d655 (0x7f6260c03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3846519Z frame #8: <unknown function> + 0xdc253 (0x7f6217c4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3846790Z frame #9: <unknown function> + 0x94ac3 (0x7f630b2ebac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3847004Z frame #10: <unknown function> + 0x126850 (0x7f630b37d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3847008Z 
2025-07-17T11:07:26.3847100Z SIGABRT(6), PID: 1805792, Thread 1806178: 
2025-07-17T11:07:26.3847573Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3847783Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3847985Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3848188Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3848687Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3848898Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3849106Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3849112Z 
2025-07-17T11:07:26.3849204Z SIGABRT(6), PID: 1805792, Thread 1806182: 
2025-07-17T11:07:26.3849671Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3849877Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3850079Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3850281Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3850778Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3850986Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3851195Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3851201Z 
2025-07-17T11:07:26.3851294Z SIGABRT(6), PID: 1805792, Thread 1806186: 
2025-07-17T11:07:26.3851760Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3851970Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3852174Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3852377Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3852877Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3853079Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3853364Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3853368Z 
2025-07-17T11:07:26.3853459Z SIGABRT(6), PID: 1805792, Thread 1806190: 
2025-07-17T11:07:26.3853993Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3854259Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3854472Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3854728Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3855228Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3855435Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3855648Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3855652Z 
2025-07-17T11:07:26.3855743Z SIGABRT(6), PID: 1805792, Thread 1806195: 
2025-07-17T11:07:26.3856271Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3856478Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3856685Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3856880Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3857380Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3857588Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3857802Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3857806Z 
2025-07-17T11:07:26.3857898Z SIGABRT(6), PID: 1805792, Thread 1806199: 
2025-07-17T11:07:26.3858372Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3858575Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3858781Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3858977Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3859472Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3859677Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3859893Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3859897Z 
2025-07-17T11:07:26.3859986Z SIGABRT(6), PID: 1805792, Thread 1806204: 
2025-07-17T11:07:26.3860460Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3860666Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3860873Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3861071Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3861654Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3861935Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3862151Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3862155Z 
2025-07-17T11:07:26.3862319Z SIGABRT(6), PID: 1805792, Thread 1806208: 
2025-07-17T11:07:26.3862792Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3863077Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3863281Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3863477Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3863976Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3864185Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3864393Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3864398Z 
2025-07-17T11:07:26.3864489Z SIGABRT(6), PID: 1805792, Thread 1806213: 
2025-07-17T11:07:26.3864959Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3865165Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3865367Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3865568Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3866062Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3866270Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3866481Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3866485Z 
2025-07-17T11:07:26.3866579Z SIGABRT(6), PID: 1805792, Thread 1806217: 
2025-07-17T11:07:26.3867048Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3867254Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3867459Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3867659Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3868153Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3868363Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3868573Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3868581Z 
2025-07-17T11:07:26.3868671Z SIGABRT(6), PID: 1805792, Thread 1806220: 
2025-07-17T11:07:26.3869140Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3869348Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3869627Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3869887Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3870386Z frame #4: <unknown function> + 0x3475ab (0x7f40a20c15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3870652Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3870943Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3870946Z 
2025-07-17T11:07:26.3871041Z SIGABRT(6), PID: 1805792, Thread 1806856: 
2025-07-17T11:07:26.3871511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3871981Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f3fb391f08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3872192Z frame #2: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3872393Z frame #3: pthread_kill + 0x12c (0x7f40a522b9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3872563Z frame #4: raise + 0x16 (0x7f40a51d7476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3872731Z frame #5: abort + 0xd3 (0x7f40a51bd7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3872975Z frame #6: <unknown function> + 0x972cc (0x7f3f214a82cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3873212Z frame #7: <unknown function> + 0x95f47 (0x7f3f214a6f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3873435Z frame #8: <unknown function> + 0x34b71 (0x7f3f21445b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3873651Z frame #9: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3873863Z frame #10: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3873867Z 
2025-07-17T11:07:26.3873964Z SIGABRT(6), PID: 1805792, Thread 1810380: 
2025-07-17T11:07:26.3874437Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3874649Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3874818Z frame #2: __poll + 0x4f (0x7f40a52adbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3875237Z frame #3: <unknown function> + 0x489f (0x7f40a4db989f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.3875470Z frame #4: <unknown function> + 0x1483bc (0x5585f1e9f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3875717Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3875943Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3876185Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3876411Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3876663Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5585f1e87a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3876895Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3877146Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5585f1e85c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3877372Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3877698Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3877982Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3878225Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3878446Z frame #16: <unknown function> + 0x147734 (0x5585f1e9e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3878732Z frame #17: <unknown function> + 0x21ca82 (0x5585f1f73a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3879011Z frame #18: <unknown function> + 0x21c9e4 (0x5585f1f739e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3879229Z frame #19: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3879442Z frame #20: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3879452Z 
2025-07-17T11:07:26.3879547Z SIGABRT(6), PID: 1805792, Thread 1810381: 
2025-07-17T11:07:26.3880030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3880241Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3880450Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3880672Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f40a52292dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3881123Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f3fdf667048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3881545Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f3fdf669bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3881780Z frame #6: <unknown function> + 0xdc253 (0x7f3fb1b8b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3881989Z frame #7: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3882201Z frame #8: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3882206Z 
2025-07-17T11:07:26.3882296Z SIGABRT(6), PID: 1805792, Thread 1810382: 
2025-07-17T11:07:26.3882769Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3882978Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3883182Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3883404Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f40a52292dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3883885Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f3fdf661741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.3884119Z frame #5: <unknown function> + 0xdc253 (0x7f3fb1b8b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3884331Z frame #6: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3884541Z frame #7: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3884546Z 
2025-07-17T11:07:26.3884639Z SIGABRT(6), PID: 1805792, Thread 1810383: 
2025-07-17T11:07:26.3885104Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3885312Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3885478Z frame #2: ioctl + 0x3f (0x7f40a52af94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3885795Z frame #3: <unknown function> + 0x13eef0 (0x7f3f2154fef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3886091Z frame #4: <unknown function> + 0x1378c4 (0x7f3f215488c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3886325Z frame #5: <unknown function> + 0x95b2b (0x7f3f214a6b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3886608Z frame #6: <unknown function> + 0x34b71 (0x7f3f21445b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3886823Z frame #7: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3887096Z frame #8: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3887100Z 
2025-07-17T11:07:26.3887192Z SIGABRT(6), PID: 1805792, Thread 1810603: 
2025-07-17T11:07:26.3887659Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3887871Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3888038Z frame #2: __poll + 0x4f (0x7f40a52adbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3888253Z frame #3: <unknown function> + 0xd4540 (0x7f3f696bf540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3888462Z frame #4: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3888670Z frame #5: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3888676Z 
2025-07-17T11:07:26.3888768Z SIGABRT(6), PID: 1805792, Thread 1810609: 
2025-07-17T11:07:26.3889242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3889447Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3889613Z frame #2: __poll + 0x4f (0x7f40a52adbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3889828Z frame #3: <unknown function> + 0xd65a2 (0x7f3f696c15a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3890038Z frame #4: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3890245Z frame #5: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3890250Z 
2025-07-17T11:07:26.3890344Z SIGABRT(6), PID: 1805792, Thread 1810617: 
2025-07-17T11:07:26.3890815Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3891024Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3891225Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3891432Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3891645Z frame #4: <unknown function> + 0xd1e7f (0x7f3f696bce7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.3891853Z frame #5: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3892063Z frame #6: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3892069Z 
2025-07-17T11:07:26.3892162Z SIGABRT(6), PID: 1805792, Thread 1810629: 
2025-07-17T11:07:26.3892631Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3892839Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3893042Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3893322Z frame #3: <unknown function> + 0x9ca53 (0x7f40a5231a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3893654Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5585f1e6bbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3893891Z frame #5: <unknown function> + 0x1d314f (0x5585f1f2a14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3894178Z frame #6: <unknown function> + 0x1d2f6b (0x5585f1f29f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3894410Z frame #7: <unknown function> + 0x6df36 (0x5585f1dc4f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3894713Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3894941Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3895182Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3895414Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3895650Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3895880Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3896169Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3896395Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3896631Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3896850Z frame #17: <unknown function> + 0x147734 (0x5585f1e9e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3897068Z frame #18: <unknown function> + 0x21ca82 (0x5585f1f73a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3897293Z frame #19: <unknown function> + 0x21c9e4 (0x5585f1f739e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3897504Z frame #20: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3897720Z frame #21: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3897724Z 
2025-07-17T11:07:26.3897815Z SIGABRT(6), PID: 1805792, Thread 1810637: 
2025-07-17T11:07:26.3898294Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3898508Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3898670Z frame #2: read + 0x4c (0x7f40a52a981c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3898850Z frame #3: _Py_read + 0x4c (0x5585f1f31a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3899071Z frame #4: <unknown function> + 0x208c3b (0x5585f1f5fc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3899296Z frame #5: <unknown function> + 0x14648d (0x5585f1e9d48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3899539Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5585f1eac215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3899762Z frame #7: <unknown function> + 0x208ae7 (0x5585f1f5fae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3899978Z frame #8: <unknown function> + 0x208472 (0x5585f1f5f472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3900202Z frame #9: <unknown function> + 0x1da6ee (0x5585f1f316ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3900430Z frame #10: <unknown function> + 0x1483bc (0x5585f1e9f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3900670Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3900895Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3901230Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5585f1e833bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3901521Z frame #14: <unknown function> + 0x147734 (0x5585f1e9e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3901773Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5585f1e85c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3902109Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3902353Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3902652Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5585f1e930ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3902888Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5585f1e837b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3903104Z frame #20: <unknown function> + 0x147734 (0x5585f1e9e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3903328Z frame #21: <unknown function> + 0x21ca82 (0x5585f1f73a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3903551Z frame #22: <unknown function> + 0x21c9e4 (0x5585f1f739e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3903762Z frame #23: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3903978Z frame #24: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3903982Z 
2025-07-17T11:07:26.3904074Z SIGABRT(6), PID: 1805792, Thread 1810670: 
2025-07-17T11:07:26.3904556Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3904770Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3904974Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3905178Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3905557Z frame #4: <unknown function> + 0x5a02023 (0x7f3fe9655023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3906133Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3fe96591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3906728Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3fe96501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3907114Z frame #7: <unknown function> + 0x79d655 (0x7f3ffab40655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3907345Z frame #8: <unknown function> + 0xdc253 (0x7f3fb1b8b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3907556Z frame #9: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3907768Z frame #10: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3907772Z 
2025-07-17T11:07:26.3907864Z SIGABRT(6), PID: 1805792, Thread 1810671: 
2025-07-17T11:07:26.3908336Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3908549Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3908750Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3908951Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3909398Z frame #4: <unknown function> + 0x5a02023 (0x7f3fe9655023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3910027Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3fe96591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3910680Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3fe96501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3911123Z frame #7: <unknown function> + 0x79d655 (0x7f3ffab40655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3911356Z frame #8: <unknown function> + 0xdc253 (0x7f3fb1b8b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3911565Z frame #9: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3911781Z frame #10: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3911785Z 
2025-07-17T11:07:26.3911878Z SIGABRT(6), PID: 1805792, Thread 1810672: 
2025-07-17T11:07:26.3912353Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3912564Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3912765Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3912966Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3913336Z frame #4: <unknown function> + 0x5a02023 (0x7f3fe9655023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3913905Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3fe96591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3914498Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3fe96501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3914879Z frame #7: <unknown function> + 0x79d655 (0x7f3ffab40655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3915107Z frame #8: <unknown function> + 0xdc253 (0x7f3fb1b8b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3915315Z frame #9: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3915533Z frame #10: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3915540Z 
2025-07-17T11:07:26.3915628Z SIGABRT(6), PID: 1805792, Thread 1810673: 
2025-07-17T11:07:26.3916104Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3916312Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3916516Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3916716Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3917089Z frame #4: <unknown function> + 0x5a02023 (0x7f3fe9655023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3917651Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3fe96591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3918242Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3fe96501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3918748Z frame #7: <unknown function> + 0x79d655 (0x7f3ffab40655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3918987Z frame #8: <unknown function> + 0xdc253 (0x7f3fb1b8b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3919255Z frame #9: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3919535Z frame #10: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3919540Z 
2025-07-17T11:07:26.3919628Z SIGABRT(6), PID: 1805792, Thread 1810674: 
2025-07-17T11:07:26.3920102Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3920310Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3920514Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3920717Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3921090Z frame #4: <unknown function> + 0x5a02023 (0x7f3fe9655023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3921650Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3fe96591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3922246Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3fe96501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3922632Z frame #7: <unknown function> + 0x79d655 (0x7f3ffab40655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3922863Z frame #8: <unknown function> + 0xdc253 (0x7f3fb1b8b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3923073Z frame #9: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3923287Z frame #10: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3923290Z 
2025-07-17T11:07:26.3923383Z SIGABRT(6), PID: 1805792, Thread 1810675: 
2025-07-17T11:07:26.3923852Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3924069Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3924269Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3924473Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3924841Z frame #4: <unknown function> + 0x5a02023 (0x7f3fe9655023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3925401Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3fe96591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3925988Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3fe96501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3926369Z frame #7: <unknown function> + 0x79d655 (0x7f3ffab40655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3926673Z frame #8: <unknown function> + 0xdc253 (0x7f3fb1b8b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3926887Z frame #9: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3927167Z frame #10: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3927172Z 
2025-07-17T11:07:26.3927268Z SIGABRT(6), PID: 1805792, Thread 1810676: 
2025-07-17T11:07:26.3927800Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3928074Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3928277Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3928478Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3928848Z frame #4: <unknown function> + 0x5a02023 (0x7f3fe9655023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3929410Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3fe96591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3929999Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3fe96501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3930385Z frame #7: <unknown function> + 0x79d655 (0x7f3ffab40655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3930615Z frame #8: <unknown function> + 0xdc253 (0x7f3fb1b8b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3930823Z frame #9: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3931035Z frame #10: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3931042Z 
2025-07-17T11:07:26.3931132Z SIGABRT(6), PID: 1805792, Thread 1810677: 
2025-07-17T11:07:26.3931604Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f3fb391eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3934875Z frame #1: <unknown function> + 0x42520 (0x7f40a51d7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3935084Z frame #2: <unknown function> + 0x91117 (0x7f40a5226117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3935284Z frame #3: pthread_cond_wait + 0x211 (0x7f40a5228a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3935657Z frame #4: <unknown function> + 0x5a02023 (0x7f3fe9655023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3936274Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f3fe96591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3936871Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f3fe96501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.3937252Z frame #7: <unknown function> + 0x79d655 (0x7f3ffab40655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3937483Z frame #8: <unknown function> + 0xdc253 (0x7f3fb1b8b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.3937694Z frame #9: <unknown function> + 0x94ac3 (0x7f40a5229ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3937910Z frame #10: <unknown function> + 0x126850 (0x7f40a52bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3937914Z 
2025-07-17T11:07:26.3938015Z ('RERUN', {'yellow': True}) [25.7483s] [100%]
2025-07-17T11:07:26.3938856Z distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph I0717 11:02:01.995000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1811294
2025-07-17T11:07:26.3939219Z I0717 11:02:01.997000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1811295
2025-07-17T11:07:26.3939646Z I0717 11:02:01.998000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1811296
2025-07-17T11:07:26.3939994Z I0717 11:02:02.034000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1811297
2025-07-17T11:07:26.3940414Z I0717 11:02:02.035000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1811298
2025-07-17T11:07:26.3940756Z I0717 11:02:02.036000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1811299
2025-07-17T11:07:26.3941107Z I0717 11:02:02.038000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1811300
2025-07-17T11:07:26.3941452Z I0717 11:02:02.039000 1800288 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1811301
2025-07-17T11:07:26.3942472Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.3942558Z   import pkg_resources
2025-07-17T11:07:26.3943537Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.3943621Z   import pkg_resources
2025-07-17T11:07:26.3944595Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.3944677Z   import pkg_resources
2025-07-17T11:07:26.3945652Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.3945732Z   import pkg_resources
2025-07-17T11:07:26.3946701Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.3946782Z   import pkg_resources
2025-07-17T11:07:26.3947751Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.3947833Z   import pkg_resources
2025-07-17T11:07:26.3948802Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.3948952Z   import pkg_resources
2025-07-17T11:07:26.3949990Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.3950131Z   import pkg_resources
2025-07-17T11:07:26.3950623Z [rank5]:W0717 11:02:23.850000 1811299 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.3951159Z [rank0]:W0717 11:02:23.851000 1811294 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.3951621Z [rank4]:W0717 11:02:23.853000 1811298 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.3952092Z [rank2]:W0717 11:02:23.854000 1811296 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.3952561Z [rank6]:W0717 11:02:23.854000 1811300 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.3953023Z [rank7]:W0717 11:02:23.854000 1811301 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.3953486Z [rank1]:W0717 11:02:23.855000 1811295 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.3953948Z [rank3]:W0717 11:02:23.855000 1811297 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.3954241Z Memory access fault by GPU node-6 (Agent handle: 0x5621e27027e0) on address 0x7fab77e23000. Reason: Unknown.
2025-07-17T11:07:26.3954514Z Memory access fault by GPU node-9 (Agent handle: 0x55f898344430) on address 0x7fdbcbe03000. Reason: Unknown.
2025-07-17T11:07:26.3954787Z Memory access fault by GPU node-3 (Agent handle: 0x5556cc511fe0) on address 0x7ff0efe05000. Reason: Unknown.
2025-07-17T11:07:26.3955054Z Memory access fault by GPU node-7 (Agent handle: 0x5556183fbe10) on address 0x7f6457e1e000. Reason: Unknown.
2025-07-17T11:07:26.3961173Z Memory access fault by GPU node-4 (Agent handle: 0x560c988c84f0) on address 0x7f14b7427000. Reason: Unknown.
2025-07-17T11:07:26.3961515Z Memory access fault by GPU node-8 (Agent handle: 0x5558d4a477a0) on address 0x7f1872803000. Reason: Unknown.
2025-07-17T11:07:26.3961808Z Memory access fault by GPU node-2 (Agent handle: 0x560616652bd0) on address 0x7f7216a03000. Reason: Unknown.
2025-07-17T11:07:26.3962088Z Memory access fault by GPU node-5 (Agent handle: 0x55b2d624bbb0) on address 0x7f6743e1d000. Reason: Unknown.
2025-07-17T11:07:26.3962194Z SIGABRT(6), PID: 1811298, Thread 1811298: 
2025-07-17T11:07:26.3962708Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3962941Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3963183Z frame #2: <unknown function> + 0x7744b (0x7fb3bd54044b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3963418Z frame #3: <unknown function> + 0x76c8e (0x7fb3bd53fc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3963647Z frame #4: <unknown function> + 0x6ad69 (0x7fb3bd533d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3963870Z frame #5: <unknown function> + 0x3f54db (0x7fb44e1984db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3964216Z frame #6: <unknown function> + 0x3e11d8 (0x7fb44e1841d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3964501Z frame #7: <unknown function> + 0xb2f86 (0x7fb44de55f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3964712Z frame #8: <unknown function> + 0x8e501 (0x7fb44de31501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3965180Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fb478c387ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.3965653Z frame #10: <unknown function> + 0xd307d9 (0x7fb49718b7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3965891Z frame #11: <unknown function> + 0x139902 (0x5621d8e29902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3966152Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5621d8e20a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3966388Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3966631Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5621d8e20a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3966865Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3967107Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3967333Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3967568Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3967795Z frame #19: <unknown function> + 0x1474b2 (0x5621d8e374b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3968001Z frame #20: PyObject_Call + 0xbc (0x5621d8e37e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3968232Z frame #21: <unknown function> + 0x1ff69d (0x5621d8eef69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3968457Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5621d8e25023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3968706Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5621d8e20cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3968937Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3969186Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5621d8e1c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3969414Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3969654Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5621d8e1c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3969876Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3970120Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5621d8e20a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3970376Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5621d8e244b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3970600Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5621d8e35ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3970821Z frame #32: <unknown function> + 0x205949 (0x5621d8ef5949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3971048Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5621d8e25023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3971289Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5621d8e20cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3971517Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3971753Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5621d8e1c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3971980Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3972295Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5621d8e1c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3972582Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3972819Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5621d8e1c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3973102Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3973348Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5621d8e1d3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3973635Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3974038Z frame #44: <unknown function> + 0x7ab3f7 (0x7fb496c063f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3974265Z frame #45: <unknown function> + 0x13bc88 (0x5621d8e2bc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3974473Z frame #46: PyObject_Call + 0x20f (0x5621d8e37f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3974723Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5621d8e2198d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3974943Z frame #48: <unknown function> + 0x147608 (0x5621d8e37608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3975187Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5621d8e1ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3975413Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3975651Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5621d8e1c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3975876Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3976232Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5621d8e1d3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3976459Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3976700Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5621d8e1c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3976926Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3977169Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5621d8e1ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3977392Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3977631Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5621d8e1ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3977849Z frame #60: <unknown function> + 0x147608 (0x5621d8e37608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3978088Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5621d8e1ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3978306Z frame #62: <unknown function> + 0x147608 (0x5621d8e37608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3978311Z 
2025-07-17T11:07:26.3978413Z SIGABRT(6), PID: 1811298, Thread 1811522: 
2025-07-17T11:07:26.3978898Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3979123Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3979332Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3979539Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3980053Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3980390Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3980678Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3980683Z 
2025-07-17T11:07:26.3980785Z SIGABRT(6), PID: 1811298, Thread 1811524: 
2025-07-17T11:07:26.3981338Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3981557Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3981837Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3982045Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3982553Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3982769Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3982982Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3982990Z 
2025-07-17T11:07:26.3983082Z SIGABRT(6), PID: 1811298, Thread 1811526: 
2025-07-17T11:07:26.3983557Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3983763Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3983967Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3984164Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3984666Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.3984876Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3985089Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3985093Z 
2025-07-17T11:07:26.3985185Z SIGABRT(6), PID: 1811301, Thread 1811301: 
2025-07-17T11:07:26.3985663Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.3985871Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.3986111Z frame #2: <unknown function> + 0x773e7 (0x7fe40fd153e7 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3986343Z frame #3: <unknown function> + 0x76c8e (0x7fe40fd14c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3986573Z frame #4: <unknown function> + 0x6ad69 (0x7fe40fd08d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.3986791Z frame #5: <unknown function> + 0x3f54db (0x7fe4a096d4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3987008Z frame #6: <unknown function> + 0x3e11d8 (0x7fe4a09591d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3987232Z frame #7: <unknown function> + 0xb2f86 (0x7fe4a062af86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3987444Z frame #8: <unknown function> + 0x8e501 (0x7fe4a0606501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.3987837Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fe4cb40d7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.3988228Z frame #10: <unknown function> + 0xd307d9 (0x7fe4e99607d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3988532Z frame #11: <unknown function> + 0x139902 (0x55f88dc35902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3988844Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55f88dc2ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3989077Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3989324Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55f88dc2ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3989608Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3989909Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3990131Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3990370Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3990594Z frame #19: <unknown function> + 0x1474b2 (0x55f88dc434b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3990800Z frame #20: PyObject_Call + 0xbc (0x55f88dc43e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3991023Z frame #21: <unknown function> + 0x1ff69d (0x55f88dcfb69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3991250Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55f88dc31023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3991492Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55f88dc2ccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3991720Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3991956Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55f88dc283bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3992178Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3992415Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55f88dc283bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3992641Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3992882Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55f88dc2ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3993139Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55f88dc304b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3993361Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55f88dc41ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3993582Z frame #32: <unknown function> + 0x205949 (0x55f88dd01949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3993805Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55f88dc31023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3994049Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55f88dc2ccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3994272Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3994511Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55f88dc283bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3994735Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3994972Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55f88dc283bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3995197Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3995435Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55f88dc283bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3995658Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3995901Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55f88dc293f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3996124Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3996591Z frame #44: <unknown function> + 0x7ab3f7 (0x7fe4e93db3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.3996891Z frame #45: <unknown function> + 0x13bc88 (0x55f88dc37c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3997103Z frame #46: PyObject_Call + 0x20f (0x55f88dc43f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3997406Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55f88dc2d98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3997687Z frame #48: <unknown function> + 0x147608 (0x55f88dc43608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3997931Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f88dc2ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3998156Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3998393Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55f88dc283bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3998615Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3998855Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55f88dc293f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3999080Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3999320Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55f88dc283bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3999544Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.3999783Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f88dc2ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4000008Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4000249Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f88dc2ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4000467Z frame #60: <unknown function> + 0x147608 (0x55f88dc43608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4000706Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f88dc2ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4000925Z frame #62: <unknown function> + 0x147608 (0x55f88dc43608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4000929Z 
2025-07-17T11:07:26.4001029Z SIGABRT(6), PID: 1811298, Thread 1811529: 
2025-07-17T11:07:26.4001511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4001727Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4001938Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4002141Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4002654Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4002862Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4003077Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4003081Z 
2025-07-17T11:07:26.4003172Z SIGABRT(6), PID: 1811301, Thread 1811679: 
2025-07-17T11:07:26.4003651Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4003855Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4004061Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4004335Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4004898Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4005118Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4005391Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4005472Z 
2025-07-17T11:07:26.4005567Z SIGABRT(6), PID: 1811298, Thread 1811531: 
2025-07-17T11:07:26.4006041Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4006247Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4006456Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4006654Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4007156Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4007364Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4007573Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4007579Z 
2025-07-17T11:07:26.4007668Z SIGABRT(6), PID: 1811301, Thread 1811683: 
2025-07-17T11:07:26.4008142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4008347Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4008551Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4008749Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4009249Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4009461Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4009673Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4009677Z 
2025-07-17T11:07:26.4009769Z SIGABRT(6), PID: 1811298, Thread 1811533: 
2025-07-17T11:07:26.4010240Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4010449Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4010653Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4010853Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4011353Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4011563Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4011770Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4011862Z SIGABRT(6), PID: 1811301, Thread 1811685: 
2025-07-17T11:07:26.4012332Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4012616Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4012880Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4013083Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4013633Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4013901Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4014113Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4014117Z 
2025-07-17T11:07:26.4014120Z 
2025-07-17T11:07:26.4014213Z SIGABRT(6), PID: 1811298, Thread 1811535: 
2025-07-17T11:07:26.4014693Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4014903Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4015106Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4015308Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4015805Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4016089Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4016298Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4016305Z 
2025-07-17T11:07:26.4016395Z SIGABRT(6), PID: 1811301, Thread 1811687: 
2025-07-17T11:07:26.4016867Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4017075Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4017280Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4017481Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4017978Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4018188Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4018397Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4018403Z 
2025-07-17T11:07:26.4018496Z SIGABRT(6), PID: 1811298, Thread 1811537: 
2025-07-17T11:07:26.4018967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4019174Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4019378Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4019578Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4020075Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4020282Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4020589Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4020593Z 
2025-07-17T11:07:26.4020682Z SIGABRT(6), PID: 1811301, Thread 1811688: 
2025-07-17T11:07:26.4021227Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4021523Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4021732Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4022004Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4022501Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4022710Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4022920Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4022924Z 
2025-07-17T11:07:26.4023015Z SIGABRT(6), PID: 1811301, Thread 1811690: 
2025-07-17T11:07:26.4023491Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4023703Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4023909Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4024104Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4024599Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4024807Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4025021Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4025026Z 
2025-07-17T11:07:26.4025115Z SIGABRT(6), PID: 1811298, Thread 1811539: 
2025-07-17T11:07:26.4025589Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4025794Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4026001Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4026199Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4026695Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4026905Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4027118Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4027122Z 
2025-07-17T11:07:26.4027210Z SIGABRT(6), PID: 1811301, Thread 1811691: 
2025-07-17T11:07:26.4027691Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4027900Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4028105Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4028305Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4028875Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4029141Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4029356Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4029360Z 
2025-07-17T11:07:26.4029509Z SIGABRT(6), PID: 1811298, Thread 1811541: 
2025-07-17T11:07:26.4029985Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4030256Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4030463Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4030661Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4031159Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4031368Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4031578Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4031582Z 
2025-07-17T11:07:26.4031674Z SIGABRT(6), PID: 1811301, Thread 1811693: 
2025-07-17T11:07:26.4032145Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4032351Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4032554Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4032757Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4033251Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4033461Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4033670Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4033675Z 
2025-07-17T11:07:26.4033766Z SIGABRT(6), PID: 1811298, Thread 1811544: 
2025-07-17T11:07:26.4034235Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4034439Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4034645Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4034845Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4035337Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4035547Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4035754Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4035760Z 
2025-07-17T11:07:26.4035851Z SIGABRT(6), PID: 1811301, Thread 1811695: 
2025-07-17T11:07:26.4036316Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4036524Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4036805Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4037066Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4037570Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4037860Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4038127Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4038132Z 
2025-07-17T11:07:26.4038224Z SIGABRT(6), PID: 1811298, Thread 1811547: 
2025-07-17T11:07:26.4038697Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4038907Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4039113Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4039313Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4039811Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4040022Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4040230Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4040236Z 
2025-07-17T11:07:26.4040325Z SIGABRT(6), PID: 1811301, Thread 1811697: 
2025-07-17T11:07:26.4040794Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4040999Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4041207Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4041405Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4041900Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4042108Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4042316Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4042320Z 
2025-07-17T11:07:26.4042409Z SIGABRT(6), PID: 1811298, Thread 1811550: 
2025-07-17T11:07:26.4042882Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4043086Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4043291Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4043488Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4043980Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4044189Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4044399Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4044564Z SIGABRT(6), PID: 1811301, Thread 1811699: 
2025-07-17T11:07:26.4045037Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4045300Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4045511Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4045767Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4046264Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4046530Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4046742Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4046748Z 
2025-07-17T11:07:26.4046751Z 
2025-07-17T11:07:26.4046841Z SIGABRT(6), PID: 1811298, Thread 1811553: 
2025-07-17T11:07:26.4047318Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4047522Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4047729Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4047924Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4048420Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4048628Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4048838Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4048927Z SIGABRT(6), PID: 1811301, Thread 1811701: 
2025-07-17T11:07:26.4049400Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4049604Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4049811Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4050010Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4050507Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4050714Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4050931Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4050935Z 
2025-07-17T11:07:26.4050939Z 
2025-07-17T11:07:26.4051030Z SIGABRT(6), PID: 1811301, Thread 1811702: 
2025-07-17T11:07:26.4051506Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4051715Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4051923Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4052124Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4052620Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4052905Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4053175Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4053270Z SIGABRT(6), PID: 1811298, Thread 1811556: 
2025-07-17T11:07:26.4053798Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4054011Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4054284Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4054485Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4054979Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4055191Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4055399Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4055403Z 
2025-07-17T11:07:26.4055406Z 
2025-07-17T11:07:26.4055497Z SIGABRT(6), PID: 1811301, Thread 1811704: 
2025-07-17T11:07:26.4056030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4056241Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4056445Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4056646Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4057141Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4057356Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4057563Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4057567Z 
2025-07-17T11:07:26.4057660Z SIGABRT(6), PID: 1811298, Thread 1811559: 
2025-07-17T11:07:26.4058129Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4058340Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4058543Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4058741Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4059241Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4059451Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4059654Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4059660Z 
2025-07-17T11:07:26.4059754Z SIGABRT(6), PID: 1811301, Thread 1811706: 
2025-07-17T11:07:26.4060227Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4060434Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4060637Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4060927Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4061500Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4061715Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4061996Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4062004Z 
2025-07-17T11:07:26.4062167Z SIGABRT(6), PID: 1811298, Thread 1811562: 
2025-07-17T11:07:26.4062639Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4062843Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4063054Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4063249Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4063748Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4063956Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4064167Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4064173Z 
2025-07-17T11:07:26.4064262Z SIGABRT(6), PID: 1811301, Thread 1811708: 
2025-07-17T11:07:26.4064732Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4064936Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4065144Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4065345Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4065842Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4066050Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4066262Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4066265Z 
2025-07-17T11:07:26.4066354Z SIGABRT(6), PID: 1811298, Thread 1811565: 
2025-07-17T11:07:26.4066826Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4067034Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4067238Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4067434Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4067932Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4068138Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4068348Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4068352Z 
2025-07-17T11:07:26.4068440Z SIGABRT(6), PID: 1811301, Thread 1811710: 
2025-07-17T11:07:26.4068914Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4069191Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4069480Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4069682Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4070240Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4070507Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4070721Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4070725Z 
2025-07-17T11:07:26.4070815Z SIGABRT(6), PID: 1811298, Thread 1811568: 
2025-07-17T11:07:26.4071294Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4071502Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4071705Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4071901Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4072402Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4072609Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4072818Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4072822Z 
2025-07-17T11:07:26.4072912Z SIGABRT(6), PID: 1811301, Thread 1811711: 
2025-07-17T11:07:26.4073382Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4073591Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4073795Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4073995Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4074486Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4074698Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4074907Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4074912Z 
2025-07-17T11:07:26.4075005Z SIGABRT(6), PID: 1811298, Thread 1811571: 
2025-07-17T11:07:26.4075476Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4075684Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4075890Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4076092Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4076590Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4076801Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4077102Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4077105Z 
2025-07-17T11:07:26.4077198Z SIGABRT(6), PID: 1811301, Thread 1811712: 
2025-07-17T11:07:26.4077728Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4077942Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4078206Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4078464Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4078962Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4079171Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4079380Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4079474Z SIGABRT(6), PID: 1811298, Thread 1811574: 
2025-07-17T11:07:26.4079947Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4080155Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4080357Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4080559Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4081054Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4081265Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4081470Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4081474Z 
2025-07-17T11:07:26.4081479Z 
2025-07-17T11:07:26.4081573Z SIGABRT(6), PID: 1811298, Thread 1811577: 
2025-07-17T11:07:26.4082043Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4082248Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4082450Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4082650Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4083141Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4083352Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4083560Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4083567Z 
2025-07-17T11:07:26.4083656Z SIGABRT(6), PID: 1811301, Thread 1811713: 
2025-07-17T11:07:26.4084126Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4084332Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4084538Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4084736Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4085236Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4085582Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4085800Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4085804Z 
2025-07-17T11:07:26.4085893Z SIGABRT(6), PID: 1811298, Thread 1811580: 
2025-07-17T11:07:26.4086439Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4086709Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4086916Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4087116Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4087622Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4087829Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4088042Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4088046Z 
2025-07-17T11:07:26.4088137Z SIGABRT(6), PID: 1811301, Thread 1811714: 
2025-07-17T11:07:26.4088612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4088819Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4089025Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4089226Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4089732Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4089940Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4090151Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4090155Z 
2025-07-17T11:07:26.4090244Z SIGABRT(6), PID: 1811298, Thread 1811583: 
2025-07-17T11:07:26.4090720Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4090925Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4091132Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4091331Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4091831Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4092037Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4092248Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4092253Z 
2025-07-17T11:07:26.4092342Z SIGABRT(6), PID: 1811301, Thread 1811715: 
2025-07-17T11:07:26.4092815Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4093021Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4093303Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4093561Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4094062Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4094330Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4094540Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4094601Z 
2025-07-17T11:07:26.4094696Z SIGABRT(6), PID: 1811298, Thread 1811586: 
2025-07-17T11:07:26.4095166Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4095378Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4095580Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4095783Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4096345Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4096556Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4096767Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4096771Z 
2025-07-17T11:07:26.4096864Z SIGABRT(6), PID: 1811301, Thread 1811716: 
2025-07-17T11:07:26.4097335Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4097543Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4097746Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4097946Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4098440Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4098651Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4098860Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4098864Z 
2025-07-17T11:07:26.4098958Z SIGABRT(6), PID: 1811298, Thread 1811589: 
2025-07-17T11:07:26.4099428Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4099638Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4099844Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4100046Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4100544Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4100753Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4100958Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4100962Z 
2025-07-17T11:07:26.4101054Z SIGABRT(6), PID: 1811301, Thread 1811717: 
2025-07-17T11:07:26.4101652Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4101939Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4102155Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4102433Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4102939Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4103229Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4103441Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4103446Z 
2025-07-17T11:07:26.4103538Z SIGABRT(6), PID: 1811298, Thread 1811592: 
2025-07-17T11:07:26.4104016Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4104227Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4104440Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4104646Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4105146Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4105358Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4105568Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4105574Z 
2025-07-17T11:07:26.4105664Z SIGABRT(6), PID: 1811301, Thread 1811718: 
2025-07-17T11:07:26.4106140Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4106346Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4106554Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4106753Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4107248Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4107458Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4107671Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4107676Z 
2025-07-17T11:07:26.4107764Z SIGABRT(6), PID: 1811298, Thread 1811595: 
2025-07-17T11:07:26.4108242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4108448Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4108652Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4108853Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4109354Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4109636Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4109848Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4109853Z 
2025-07-17T11:07:26.4110002Z SIGABRT(6), PID: 1811301, Thread 1811719: 
2025-07-17T11:07:26.4110535Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4110743Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4111012Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4111209Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4111707Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4111916Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4112130Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4112135Z 
2025-07-17T11:07:26.4112223Z SIGABRT(6), PID: 1811298, Thread 1811598: 
2025-07-17T11:07:26.4112696Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4112903Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4113108Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4113306Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4113802Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4114011Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4114223Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4114227Z 
2025-07-17T11:07:26.4114318Z SIGABRT(6), PID: 1811301, Thread 1811720: 
2025-07-17T11:07:26.4114799Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4115011Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4115215Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4115418Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4115929Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4116144Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4116354Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4116449Z SIGABRT(6), PID: 1811298, Thread 1811602: 
2025-07-17T11:07:26.4116923Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4117132Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4117336Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4117536Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4118175Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4118388Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4118598Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4118663Z 
2025-07-17T11:07:26.4118667Z 
2025-07-17T11:07:26.4118763Z SIGABRT(6), PID: 1811298, Thread 1811605: 
2025-07-17T11:07:26.4119296Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4119503Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4119705Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4119908Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4120401Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4120614Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4120824Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4120830Z 
2025-07-17T11:07:26.4120922Z SIGABRT(6), PID: 1811301, Thread 1811721: 
2025-07-17T11:07:26.4121394Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4121602Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4121809Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4122011Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4122517Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4122739Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4122952Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4122957Z 
2025-07-17T11:07:26.4123050Z SIGABRT(6), PID: 1811301, Thread 1811722: 
2025-07-17T11:07:26.4123531Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4123741Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4123949Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4124151Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4124666Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4124882Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4125096Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4125192Z SIGABRT(6), PID: 1811298, Thread 1811608: 
2025-07-17T11:07:26.4125669Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4125957Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4126165Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4126427Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4126988Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4127203Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4127494Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4127500Z 
2025-07-17T11:07:26.4127503Z 
2025-07-17T11:07:26.4127595Z SIGABRT(6), PID: 1811298, Thread 1811611: 
2025-07-17T11:07:26.4128071Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4128279Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4128488Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4128686Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4129188Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4129395Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4129605Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4129695Z SIGABRT(6), PID: 1811301, Thread 1811723: 
2025-07-17T11:07:26.4130173Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4130380Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4130585Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4130781Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4131282Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4131489Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4131701Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4131705Z 
2025-07-17T11:07:26.4131710Z 
2025-07-17T11:07:26.4131798Z SIGABRT(6), PID: 1811301, Thread 1811724: 
2025-07-17T11:07:26.4132276Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4132481Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4132687Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4132888Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4133384Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4133593Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4133807Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4133881Z 
2025-07-17T11:07:26.4133973Z SIGABRT(6), PID: 1811298, Thread 1811614: 
2025-07-17T11:07:26.4134507Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4134714Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4134981Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4135181Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4135738Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4136007Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4136223Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4136228Z 
2025-07-17T11:07:26.4136316Z SIGABRT(6), PID: 1811301, Thread 1811725: 
2025-07-17T11:07:26.4136789Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4136998Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4137202Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4137405Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4137898Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4138107Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4138316Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4138320Z 
2025-07-17T11:07:26.4138413Z SIGABRT(6), PID: 1811298, Thread 1811617: 
2025-07-17T11:07:26.4138886Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4139094Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4139297Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4139498Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4139993Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4140207Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4140415Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4140419Z 
2025-07-17T11:07:26.4140510Z SIGABRT(6), PID: 1811301, Thread 1811726: 
2025-07-17T11:07:26.4140983Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4141191Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4141395Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4141594Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4142086Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4142501Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4142788Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4142792Z 
2025-07-17T11:07:26.4142888Z SIGABRT(6), PID: 1811298, Thread 1811620: 
2025-07-17T11:07:26.4143427Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4143707Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4143915Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4144115Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4144611Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4144821Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4145029Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4145032Z 
2025-07-17T11:07:26.4145128Z SIGABRT(6), PID: 1811298, Thread 1811623: 
2025-07-17T11:07:26.4145605Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4145814Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4146021Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4146221Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4146718Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4146929Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4147136Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4147229Z SIGABRT(6), PID: 1811301, Thread 1811727: 
2025-07-17T11:07:26.4147704Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4147912Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4148114Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4148316Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4148813Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4149019Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4149230Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4149234Z 
2025-07-17T11:07:26.4149237Z 
2025-07-17T11:07:26.4149326Z SIGABRT(6), PID: 1811298, Thread 1811625: 
2025-07-17T11:07:26.4149801Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4150003Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4150285Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4150483Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4151040Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4151306Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4151522Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4151604Z 
2025-07-17T11:07:26.4151695Z SIGABRT(6), PID: 1811301, Thread 1811728: 
2025-07-17T11:07:26.4152173Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4152377Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4152585Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4152784Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4153280Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4153488Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4153701Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4153789Z SIGABRT(6), PID: 1811298, Thread 1811628: 
2025-07-17T11:07:26.4154265Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4154471Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4154676Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4154873Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4155372Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4155578Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4155789Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4155793Z 
2025-07-17T11:07:26.4155796Z 
2025-07-17T11:07:26.4155884Z SIGABRT(6), PID: 1811298, Thread 1811631: 
2025-07-17T11:07:26.4156362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4156565Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4156774Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4156970Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4157467Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4157677Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4157887Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4157890Z 
2025-07-17T11:07:26.4157978Z SIGABRT(6), PID: 1811301, Thread 1811729: 
2025-07-17T11:07:26.4158528Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4158799Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4159013Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4159296Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4159802Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4160080Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4160291Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4160295Z 
2025-07-17T11:07:26.4160390Z SIGABRT(6), PID: 1811298, Thread 1811633: 
2025-07-17T11:07:26.4160864Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4161076Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4161282Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4161485Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4161981Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4162191Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4162398Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4162404Z 
2025-07-17T11:07:26.4162498Z SIGABRT(6), PID: 1811301, Thread 1811730: 
2025-07-17T11:07:26.4162975Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4163183Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4163389Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4163588Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4164088Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4164297Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4164504Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4164508Z 
2025-07-17T11:07:26.4164599Z SIGABRT(6), PID: 1811298, Thread 1811635: 
2025-07-17T11:07:26.4165073Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4165283Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4165489Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4165693Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4166189Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4166398Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4166680Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4166683Z 
2025-07-17T11:07:26.4166835Z SIGABRT(6), PID: 1811301, Thread 1811731: 
2025-07-17T11:07:26.4167311Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4167581Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4167845Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4168046Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4168544Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4168755Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4168967Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4168979Z 
2025-07-17T11:07:26.4169066Z SIGABRT(6), PID: 1811298, Thread 1811638: 
2025-07-17T11:07:26.4169546Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4169753Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4169960Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4170156Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4170661Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4170872Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4171085Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4171090Z 
2025-07-17T11:07:26.4171180Z SIGABRT(6), PID: 1811301, Thread 1811732: 
2025-07-17T11:07:26.4171657Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4171862Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4172068Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4172263Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4172767Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4172978Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4173192Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4173196Z 
2025-07-17T11:07:26.4173287Z SIGABRT(6), PID: 1811298, Thread 1811641: 
2025-07-17T11:07:26.4173760Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4173966Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4174171Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4174370Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4174941Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4175212Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4175431Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4175435Z 
2025-07-17T11:07:26.4175583Z SIGABRT(6), PID: 1811301, Thread 1811733: 
2025-07-17T11:07:26.4176145Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4176442Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4176654Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4176860Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4177371Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4177582Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4177798Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4177803Z 
2025-07-17T11:07:26.4177893Z SIGABRT(6), PID: 1811298, Thread 1811644: 
2025-07-17T11:07:26.4178373Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4178581Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4178788Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4178986Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4179490Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4179699Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4179909Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4179915Z 
2025-07-17T11:07:26.4180006Z SIGABRT(6), PID: 1811301, Thread 1811734: 
2025-07-17T11:07:26.4180478Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4180693Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4180898Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4181099Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4181598Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4181810Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4182020Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4182024Z 
2025-07-17T11:07:26.4182115Z SIGABRT(6), PID: 1811298, Thread 1811647: 
2025-07-17T11:07:26.4182585Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4182882Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4183090Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4183367Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4183937Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4184153Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4184431Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4184435Z 
2025-07-17T11:07:26.4184527Z SIGABRT(6), PID: 1811301, Thread 1811735: 
2025-07-17T11:07:26.4185004Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4185217Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4185422Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4185624Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4186121Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4186336Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4186542Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4186546Z 
2025-07-17T11:07:26.4186637Z SIGABRT(6), PID: 1811298, Thread 1811649: 
2025-07-17T11:07:26.4187108Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4187322Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4187525Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4187726Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4188221Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4188437Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4188642Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4188646Z 
2025-07-17T11:07:26.4188738Z SIGABRT(6), PID: 1811301, Thread 1811736: 
2025-07-17T11:07:26.4189214Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4189424Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4189628Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4189832Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4190326Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4190538Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4190747Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4190821Z 
2025-07-17T11:07:26.4190912Z SIGABRT(6), PID: 1811298, Thread 1811652: 
2025-07-17T11:07:26.4191468Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4191677Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4191940Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4192141Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4192708Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4192915Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4193127Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4193133Z 
2025-07-17T11:07:26.4193222Z SIGABRT(6), PID: 1811301, Thread 1811737: 
2025-07-17T11:07:26.4193696Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4193904Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4194111Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4194311Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4194810Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4195018Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4195229Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4195233Z 
2025-07-17T11:07:26.4195321Z SIGABRT(6), PID: 1811301, Thread 1811738: 
2025-07-17T11:07:26.4195797Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4196005Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4196209Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4196406Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4196901Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4197110Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4197321Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4197413Z SIGABRT(6), PID: 1811298, Thread 1811653: 
2025-07-17T11:07:26.4197888Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4198095Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4198304Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4198502Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4199002Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4199282Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4199555Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4199559Z 
2025-07-17T11:07:26.4199562Z 
2025-07-17T11:07:26.4199654Z SIGABRT(6), PID: 1811301, Thread 1811739: 
2025-07-17T11:07:26.4200186Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4200452Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4200656Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4200853Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4201349Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4201560Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4201768Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4201771Z 
2025-07-17T11:07:26.4201860Z SIGABRT(6), PID: 1811298, Thread 1811655: 
2025-07-17T11:07:26.4202328Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4202539Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4202741Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4202942Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4203438Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4203650Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4203859Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4203863Z 
2025-07-17T11:07:26.4203956Z SIGABRT(6), PID: 1811301, Thread 1811740: 
2025-07-17T11:07:26.4204424Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4204635Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4204837Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4205037Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4205538Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4205747Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4205958Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4205962Z 
2025-07-17T11:07:26.4206051Z SIGABRT(6), PID: 1811298, Thread 1811657: 
2025-07-17T11:07:26.4206527Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4206736Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4206938Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4207214Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4207776Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4207988Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4208254Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4208315Z 
2025-07-17T11:07:26.4208409Z SIGABRT(6), PID: 1811301, Thread 1811741: 
2025-07-17T11:07:26.4208881Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4209088Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4209294Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4209497Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4209991Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4210199Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4210407Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4210501Z SIGABRT(6), PID: 1811298, Thread 1811659: 
2025-07-17T11:07:26.4210972Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4211179Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4211383Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4211581Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4212074Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4212287Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4212497Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4212505Z 
2025-07-17T11:07:26.4212509Z 
2025-07-17T11:07:26.4212598Z SIGABRT(6), PID: 1811301, Thread 1811742: 
2025-07-17T11:07:26.4213071Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4213279Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4213487Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4213688Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4214189Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4214399Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4214610Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4214613Z 
2025-07-17T11:07:26.4214702Z SIGABRT(6), PID: 1811298, Thread 1811661: 
2025-07-17T11:07:26.4215173Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4215454Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4215720Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4215972Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4216550Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4216836Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4217046Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4217050Z 
2025-07-17T11:07:26.4217138Z SIGABRT(6), PID: 1811301, Thread 1811743: 
2025-07-17T11:07:26.4217617Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4217824Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4218029Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4218229Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4218733Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4218940Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4219149Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4219154Z 
2025-07-17T11:07:26.4219242Z SIGABRT(6), PID: 1811298, Thread 1811663: 
2025-07-17T11:07:26.4219721Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4219926Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4220133Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4220333Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4220834Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4221039Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4221249Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4221254Z 
2025-07-17T11:07:26.4221345Z SIGABRT(6), PID: 1811301, Thread 1811744: 
2025-07-17T11:07:26.4221824Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4222032Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4222238Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4222439Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4222938Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4223146Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4223461Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4223465Z 
2025-07-17T11:07:26.4223557Z SIGABRT(6), PID: 1811298, Thread 1811667: 
2025-07-17T11:07:26.4224105Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4224377Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4224583Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4224843Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4225337Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4225548Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4225755Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4225759Z 
2025-07-17T11:07:26.4225856Z SIGABRT(6), PID: 1811298, Thread 1811669: 
2025-07-17T11:07:26.4226323Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4226536Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4226738Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4226937Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4227433Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4227646Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4227856Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4227860Z 
2025-07-17T11:07:26.4227950Z SIGABRT(6), PID: 1811301, Thread 1811745: 
2025-07-17T11:07:26.4228422Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4228630Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4228834Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4229037Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4229530Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4229741Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4229949Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4229953Z 
2025-07-17T11:07:26.4230047Z SIGABRT(6), PID: 1811298, Thread 1811671: 
2025-07-17T11:07:26.4230518Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4230730Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4230934Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4231131Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4231703Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4231980Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4232192Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4232198Z 
2025-07-17T11:07:26.4232373Z SIGABRT(6), PID: 1811301, Thread 1811746: 
2025-07-17T11:07:26.4232851Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4233120Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4233327Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4233523Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4234024Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4234231Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4234449Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4234455Z 
2025-07-17T11:07:26.4234543Z SIGABRT(6), PID: 1811298, Thread 1811673: 
2025-07-17T11:07:26.4235030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4235234Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4235440Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4235638Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4236137Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4236344Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4236557Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4236562Z 
2025-07-17T11:07:26.4236652Z SIGABRT(6), PID: 1811301, Thread 1811747: 
2025-07-17T11:07:26.4237131Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4237336Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4237546Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4237745Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4238247Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4238455Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4238665Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4238671Z 
2025-07-17T11:07:26.4238759Z SIGABRT(6), PID: 1811298, Thread 1811675: 
2025-07-17T11:07:26.4239237Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4239442Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4239720Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4239975Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4240476Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4240743Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4241011Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4241015Z 
2025-07-17T11:07:26.4241105Z SIGABRT(6), PID: 1811301, Thread 1811748: 
2025-07-17T11:07:26.4241580Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4241786Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4241995Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4242189Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4242693Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4242900Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4243115Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4243118Z 
2025-07-17T11:07:26.4243207Z SIGABRT(6), PID: 1811298, Thread 1811676: 
2025-07-17T11:07:26.4243681Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4243894Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4244100Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4244299Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4244796Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4245004Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4245215Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4245219Z 
2025-07-17T11:07:26.4245310Z SIGABRT(6), PID: 1811301, Thread 1811749: 
2025-07-17T11:07:26.4245784Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4245995Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4246202Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4246402Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4246898Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4247107Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4247314Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4247318Z 
2025-07-17T11:07:26.4247488Z SIGABRT(6), PID: 1811298, Thread 1811678: 
2025-07-17T11:07:26.4247960Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4248232Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4248439Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4248699Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4249272Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4249480Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4249687Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4249693Z 
2025-07-17T11:07:26.4249786Z SIGABRT(6), PID: 1811301, Thread 1811750: 
2025-07-17T11:07:26.4250259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4250467Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4250669Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4250875Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4251374Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4251584Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4251793Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4251889Z SIGABRT(6), PID: 1811298, Thread 1811681: 
2025-07-17T11:07:26.4252366Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4252573Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4252777Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4252978Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4253475Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4253684Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4253892Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4253899Z 
2025-07-17T11:07:26.4253903Z 
2025-07-17T11:07:26.4253994Z SIGABRT(6), PID: 1811298, Thread 1811684: 
2025-07-17T11:07:26.4254476Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4254683Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4254892Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4255089Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4255585Z frame #4: <unknown function> + 0x3475ab (0x7fb53e1785ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4255870Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4256214Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4256219Z 
2025-07-17T11:07:26.4256313Z SIGABRT(6), PID: 1811301, Thread 1811751: 
2025-07-17T11:07:26.4256861Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4257069Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4257353Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4257555Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4258053Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4258262Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4258475Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4258479Z 
2025-07-17T11:07:26.4258567Z SIGABRT(6), PID: 1811298, Thread 1812361: 
2025-07-17T11:07:26.4259043Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4259497Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fb44f9d708a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4259706Z frame #2: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4259899Z frame #3: pthread_kill + 0x12c (0x7fb5412e29fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4260073Z frame #4: raise + 0x16 (0x7fb54128e476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4260233Z frame #5: abort + 0xd3 (0x7fb5412747f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4260478Z frame #6: <unknown function> + 0x972cc (0x7fb3bd5602cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4260710Z frame #7: <unknown function> + 0x95f47 (0x7fb3bd55ef47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4260942Z frame #8: <unknown function> + 0x34b71 (0x7fb3bd4fdb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4261150Z frame #9: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4261366Z frame #10: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4261370Z 
2025-07-17T11:07:26.4261462Z SIGABRT(6), PID: 1811301, Thread 1811752: 
2025-07-17T11:07:26.4261941Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4262154Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4262365Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4262567Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4263075Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4263283Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4263496Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4263500Z 
2025-07-17T11:07:26.4263589Z SIGABRT(6), PID: 1811298, Thread 1815886: 
2025-07-17T11:07:26.4264154Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4264421Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4264595Z frame #2: __poll + 0x4f (0x7fb541364bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4265063Z frame #3: <unknown function> + 0x489f (0x7fb540e7089f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.4265469Z frame #4: <unknown function> + 0x1483bc (0x5621d8e383bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4265716Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4265947Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4266182Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4266410Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4266658Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5621d8e20a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4266895Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4267146Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5621d8e1ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4267377Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4267611Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4267839Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4268073Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4268296Z frame #16: <unknown function> + 0x147734 (0x5621d8e37734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4268515Z frame #17: <unknown function> + 0x21ca82 (0x5621d8f0ca82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4268738Z frame #18: <unknown function> + 0x21c9e4 (0x5621d8f0c9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4268951Z frame #19: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4269165Z frame #20: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4269172Z 
2025-07-17T11:07:26.4269266Z SIGABRT(6), PID: 1811301, Thread 1811753: 
2025-07-17T11:07:26.4269746Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4269963Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4270171Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4270373Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4270881Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4271097Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4271307Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4271312Z 
2025-07-17T11:07:26.4271404Z SIGABRT(6), PID: 1811301, Thread 1811754: 
2025-07-17T11:07:26.4271872Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4272159Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4272423Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4272631Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4273186Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4273457Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4273667Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4273671Z 
2025-07-17T11:07:26.4273763Z SIGABRT(6), PID: 1811301, Thread 1811755: 
2025-07-17T11:07:26.4274230Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4274442Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4274647Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4274846Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4275341Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4275553Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4275760Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4275764Z 
2025-07-17T11:07:26.4275857Z SIGABRT(6), PID: 1811298, Thread 1815887: 
2025-07-17T11:07:26.4276328Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4276536Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4276742Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4276974Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fb5412e02dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4277422Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fb47b71f048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.4277856Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fb47b721bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.4278086Z frame #6: <unknown function> + 0xdc253 (0x7fb44dc43253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4278300Z frame #7: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4278510Z frame #8: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4278514Z 
2025-07-17T11:07:26.4278609Z SIGABRT(6), PID: 1811301, Thread 1811756: 
2025-07-17T11:07:26.4279081Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4279294Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4279500Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4279703Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4280206Z frame #4: <unknown function> + 0x3475ab (0x7fe5909505ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4280489Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4280758Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4280766Z 
2025-07-17T11:07:26.4280858Z SIGABRT(6), PID: 1811301, Thread 1812356: 
2025-07-17T11:07:26.4281409Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4281925Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fe4a21ac08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4282133Z frame #2: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4282327Z frame #3: pthread_kill + 0x12c (0x7fe593aba9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4282497Z frame #4: raise + 0x16 (0x7fe593a66476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4282659Z frame #5: abort + 0xd3 (0x7fe593a4c7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4282896Z frame #6: <unknown function> + 0x972cc (0x7fe40fd352cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4283127Z frame #7: <unknown function> + 0x95f47 (0x7fe40fd33f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4283356Z frame #8: <unknown function> + 0x34b71 (0x7fe40fcd2b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4283566Z frame #9: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4283782Z frame #10: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4283786Z 
2025-07-17T11:07:26.4283875Z SIGABRT(6), PID: 1811298, Thread 1815888: 
2025-07-17T11:07:26.4284351Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4284562Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4284777Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4284998Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fb5412e02dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4285473Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fb47b719741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.4285706Z frame #5: <unknown function> + 0xdc253 (0x7fb44dc43253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4285916Z frame #6: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4286125Z frame #7: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4286131Z 
2025-07-17T11:07:26.4286223Z SIGABRT(6), PID: 1811298, Thread 1815891: 
2025-07-17T11:07:26.4286696Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4286904Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4287069Z frame #2: ioctl + 0x3f (0x7fb54136694f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4287305Z frame #3: <unknown function> + 0x13eef0 (0x7fb3bd607ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4287536Z frame #4: <unknown function> + 0x1378c4 (0x7fb3bd6008c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4287766Z frame #5: <unknown function> + 0x95b2b (0x7fb3bd55eb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4287991Z frame #6: <unknown function> + 0x34b71 (0x7fb3bd4fdb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4288276Z frame #7: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4288546Z frame #8: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4288550Z 
2025-07-17T11:07:26.4288649Z SIGABRT(6), PID: 1811301, Thread 1814876: 
2025-07-17T11:07:26.4289175Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4289388Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4289622Z frame #2: __poll + 0x4f (0x7fe593b3cbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4290033Z frame #3: <unknown function> + 0x489f (0x7fe59364889f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.4290262Z frame #4: <unknown function> + 0x1483bc (0x55f88dc443bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4290511Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4290741Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4290982Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4291207Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4291458Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55f88dc2ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4291690Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4291941Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f88dc2ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4292167Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4292410Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4292634Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4292869Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4293087Z frame #16: <unknown function> + 0x147734 (0x55f88dc43734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4293313Z frame #17: <unknown function> + 0x21ca82 (0x55f88dd18a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4293534Z frame #18: <unknown function> + 0x21c9e4 (0x55f88dd189e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4293750Z frame #19: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4293961Z frame #20: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4293967Z 
2025-07-17T11:07:26.4294059Z SIGABRT(6), PID: 1811298, Thread 1816110: 
2025-07-17T11:07:26.4294531Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4294745Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4294910Z frame #2: __poll + 0x4f (0x7fb541364bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4295125Z frame #3: <unknown function> + 0xd4540 (0x7fb405777540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.4295333Z frame #4: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4295542Z frame #5: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4295547Z 
2025-07-17T11:07:26.4295636Z SIGABRT(6), PID: 1811298, Thread 1816116: 
2025-07-17T11:07:26.4296244Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4296530Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4296699Z frame #2: __poll + 0x4f (0x7fb541364bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4296982Z frame #3: <unknown function> + 0xd65a2 (0x7fb4057795a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.4297194Z frame #4: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4297478Z frame #5: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4297482Z 
2025-07-17T11:07:26.4297576Z SIGABRT(6), PID: 1811298, Thread 1816124: 
2025-07-17T11:07:26.4298048Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4298261Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4298466Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4298677Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4298890Z frame #4: <unknown function> + 0xd1e7f (0x7fb405774e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.4299100Z frame #5: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4299307Z frame #6: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4299312Z 
2025-07-17T11:07:26.4299403Z SIGABRT(6), PID: 1811301, Thread 1814877: 
2025-07-17T11:07:26.4299868Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4300079Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4300282Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4300509Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fe593ab82dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4300952Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fe4cdef4048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.4301378Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fe4cdef6bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.4301610Z frame #6: <unknown function> + 0xdc253 (0x7fe4a0418253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4301820Z frame #7: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4302036Z frame #8: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4302041Z 
2025-07-17T11:07:26.4302129Z SIGABRT(6), PID: 1811298, Thread 1816134: 
2025-07-17T11:07:26.4302609Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4302819Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4303022Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4303228Z frame #3: <unknown function> + 0x9ca53 (0x7fb5412e8a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4303481Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5621d8e04bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4303705Z frame #5: <unknown function> + 0x1d314f (0x5621d8ec314f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4304024Z frame #6: <unknown function> + 0x1d2f6b (0x5621d8ec2f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4304307Z frame #7: <unknown function> + 0x6df36 (0x5621d8d5df36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4304551Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4304776Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4305082Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4305370Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4305610Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4305835Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4306076Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4306299Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4306538Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4306761Z frame #17: <unknown function> + 0x147734 (0x5621d8e37734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4306985Z frame #18: <unknown function> + 0x21ca82 (0x5621d8f0ca82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4307207Z frame #19: <unknown function> + 0x21c9e4 (0x5621d8f0c9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4307425Z frame #20: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4307636Z frame #21: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4307642Z 
2025-07-17T11:07:26.4307738Z SIGABRT(6), PID: 1811301, Thread 1814878: 
2025-07-17T11:07:26.4308216Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4308430Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4308635Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4308863Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fe593ab82dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4309345Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fe4cdeee741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.4309571Z frame #5: <unknown function> + 0xdc253 (0x7fe4a0418253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4309779Z frame #6: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4309990Z frame #7: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4309994Z 
2025-07-17T11:07:26.4310085Z SIGABRT(6), PID: 1811301, Thread 1814882: 
2025-07-17T11:07:26.4310565Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4310771Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4310939Z frame #2: ioctl + 0x3f (0x7fe593b3e94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4311175Z frame #3: <unknown function> + 0x13eef0 (0x7fe40fddcef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4311407Z frame #4: <unknown function> + 0x1378c4 (0x7fe40fdd58c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4311634Z frame #5: <unknown function> + 0x95b2b (0x7fe40fd33b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4311944Z frame #6: <unknown function> + 0x34b71 (0x7fe40fcd2b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4312257Z frame #7: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4312473Z frame #8: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4312477Z 
2025-07-17T11:07:26.4312571Z SIGABRT(6), PID: 1811298, Thread 1816151: 
2025-07-17T11:07:26.4313109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4313376Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4313541Z frame #2: read + 0x4c (0x7fb54136081c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4313722Z frame #3: _Py_read + 0x4c (0x5621d8ecaa6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4313949Z frame #4: <unknown function> + 0x208c3b (0x5621d8ef8c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4314169Z frame #5: <unknown function> + 0x14648d (0x5621d8e3648d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4314416Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5621d8e45215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4314637Z frame #7: <unknown function> + 0x208ae7 (0x5621d8ef8ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4314857Z frame #8: <unknown function> + 0x208472 (0x5621d8ef8472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4315087Z frame #9: <unknown function> + 0x1da6ee (0x5621d8eca6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4315310Z frame #10: <unknown function> + 0x1483bc (0x5621d8e383bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4315552Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4315781Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4316018Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5621d8e1c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4316239Z frame #14: <unknown function> + 0x147734 (0x5621d8e37734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4316487Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5621d8e1ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4316715Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4316950Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4317176Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5621d8e2c0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4317412Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5621d8e1c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4317636Z frame #20: <unknown function> + 0x147734 (0x5621d8e37734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4317856Z frame #21: <unknown function> + 0x21ca82 (0x5621d8f0ca82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4318077Z frame #22: <unknown function> + 0x21c9e4 (0x5621d8f0c9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4318288Z frame #23: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4318506Z frame #24: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4318511Z 
2025-07-17T11:07:26.4318602Z SIGABRT(6), PID: 1811301, Thread 1816106: 
2025-07-17T11:07:26.4319080Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4319290Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4319535Z frame #2: __poll + 0x4f (0x7fe593b3cbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4319750Z frame #3: <unknown function> + 0xd4540 (0x7fe457f4c540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.4320020Z frame #4: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4320230Z frame #5: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4320234Z 
2025-07-17T11:07:26.4320383Z SIGABRT(6), PID: 1811301, Thread 1816112: 
2025-07-17T11:07:26.4320856Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4321125Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4321290Z frame #2: __poll + 0x4f (0x7fe593b3cbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4321506Z frame #3: <unknown function> + 0xd65a2 (0x7fe457f4e5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.4321714Z frame #4: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4327673Z frame #5: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4327686Z 
2025-07-17T11:07:26.4327822Z SIGABRT(6), PID: 1811301, Thread 1816123: 
2025-07-17T11:07:26.4328375Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4328610Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4328824Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4329032Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4329257Z frame #4: <unknown function> + 0xd1e7f (0x7fe457f49e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.4329467Z frame #5: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4329680Z frame #6: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4329685Z 
2025-07-17T11:07:26.4329780Z SIGABRT(6), PID: 1811301, Thread 1816130: 
2025-07-17T11:07:26.4330274Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4330489Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4330696Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4330899Z frame #3: <unknown function> + 0x9ca53 (0x7fe593ac0a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4331156Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55f88dc10bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4331390Z frame #5: <unknown function> + 0x1d314f (0x55f88dccf14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4331616Z frame #6: <unknown function> + 0x1d2f6b (0x55f88dccef6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4331835Z frame #7: <unknown function> + 0x6df36 (0x55f88db69f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4332084Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4332305Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4332551Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4332776Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4333014Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4333360Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4333663Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4333890Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4334185Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4334407Z frame #17: <unknown function> + 0x147734 (0x55f88dc43734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4334694Z frame #18: <unknown function> + 0x21ca82 (0x55f88dd18a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4334910Z frame #19: <unknown function> + 0x21c9e4 (0x55f88dd189e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4335123Z frame #20: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4335334Z frame #21: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4335342Z 
2025-07-17T11:07:26.4335434Z SIGABRT(6), PID: 1811301, Thread 1816140: 
2025-07-17T11:07:26.4336037Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4336252Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4336417Z frame #2: read + 0x4c (0x7fe593b3881c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4336602Z frame #3: _Py_read + 0x4c (0x55f88dcd6a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4336824Z frame #4: <unknown function> + 0x208c3b (0x55f88dd04c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4337043Z frame #5: <unknown function> + 0x14648d (0x55f88dc4248d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4337286Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55f88dc51215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4337504Z frame #7: <unknown function> + 0x208ae7 (0x55f88dd04ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4337717Z frame #8: <unknown function> + 0x208472 (0x55f88dd04472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4337933Z frame #9: <unknown function> + 0x1da6ee (0x55f88dcd66ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4338154Z frame #10: <unknown function> + 0x1483bc (0x55f88dc443bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4338393Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4338614Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4338849Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55f88dc283bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4339062Z frame #14: <unknown function> + 0x147734 (0x55f88dc43734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4339309Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f88dc2ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4339532Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4339765Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4339984Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55f88dc380ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4340218Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55f88dc287b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4340431Z frame #20: <unknown function> + 0x147734 (0x55f88dc43734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4340647Z frame #21: <unknown function> + 0x21ca82 (0x55f88dd18a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4340963Z frame #22: <unknown function> + 0x21c9e4 (0x55f88dd189e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4341178Z frame #23: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4341456Z frame #24: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4341464Z 
2025-07-17T11:07:26.4341558Z SIGABRT(6), PID: 1811298, Thread 1816208: 
2025-07-17T11:07:26.4342113Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4342400Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4342606Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4342807Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4343188Z frame #4: <unknown function> + 0x5a02023 (0x7fb48570d023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4343756Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb4857111ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4344355Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb4857081d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4344745Z frame #7: <unknown function> + 0x79d655 (0x7fb496bf8655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4344983Z frame #8: <unknown function> + 0xdc253 (0x7fb44dc43253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4345197Z frame #9: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4345414Z frame #10: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4345419Z 
2025-07-17T11:07:26.4345512Z SIGABRT(6), PID: 1811301, Thread 1816160: 
2025-07-17T11:07:26.4345996Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4346205Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4346413Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4346614Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4346989Z frame #4: <unknown function> + 0x5a02023 (0x7fe4d7ee2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4347546Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe4d7ee61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4348137Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe4d7edd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4348514Z frame #7: <unknown function> + 0x79d655 (0x7fe4e93cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4348747Z frame #8: <unknown function> + 0xdc253 (0x7fe4a0418253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4348953Z frame #9: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4349166Z frame #10: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4349170Z 
2025-07-17T11:07:26.4349261Z SIGABRT(6), PID: 1811298, Thread 1816209: 
2025-07-17T11:07:26.4349811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4350077Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4350289Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4350547Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4350921Z frame #4: <unknown function> + 0x5a02023 (0x7fb48570d023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4351549Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb4857111ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4352143Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb4857081d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4352526Z frame #7: <unknown function> + 0x79d655 (0x7fb496bf8655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4352751Z frame #8: <unknown function> + 0xdc253 (0x7fb44dc43253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4352960Z frame #9: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4353171Z frame #10: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4353175Z 
2025-07-17T11:07:26.4353267Z SIGABRT(6), PID: 1811301, Thread 1816161: 
2025-07-17T11:07:26.4353736Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4353942Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4354142Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4354344Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4354716Z frame #4: <unknown function> + 0x5a02023 (0x7fe4d7ee2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4355275Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe4d7ee61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4355859Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe4d7edd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4356238Z frame #7: <unknown function> + 0x79d655 (0x7fe4e93cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4356465Z frame #8: <unknown function> + 0xdc253 (0x7fe4a0418253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4356673Z frame #9: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4356885Z frame #10: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4356889Z 
2025-07-17T11:07:26.4356982Z SIGABRT(6), PID: 1811298, Thread 1816210: 
2025-07-17T11:07:26.4357452Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4357661Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4357865Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4358141Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4358569Z frame #4: <unknown function> + 0x5a02023 (0x7fb48570d023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4359211Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb4857111ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4359798Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb4857081d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4360243Z frame #7: <unknown function> + 0x79d655 (0x7fb496bf8655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4360475Z frame #8: <unknown function> + 0xdc253 (0x7fb44dc43253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4360682Z frame #9: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4360896Z frame #10: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4360900Z 
2025-07-17T11:07:26.4360987Z SIGABRT(6), PID: 1811301, Thread 1816162: 
2025-07-17T11:07:26.4361458Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4361664Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4361866Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4362063Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4362432Z frame #4: <unknown function> + 0x5a02023 (0x7fe4d7ee2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4362987Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe4d7ee61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4363576Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe4d7edd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4363954Z frame #7: <unknown function> + 0x79d655 (0x7fe4e93cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4364179Z frame #8: <unknown function> + 0xdc253 (0x7fe4a0418253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4364383Z frame #9: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4364598Z frame #10: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4364603Z 
2025-07-17T11:07:26.4364691Z SIGABRT(6), PID: 1811298, Thread 1816211: 
2025-07-17T11:07:26.4365162Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4365366Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4365572Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4365770Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4366142Z frame #4: <unknown function> + 0x5a02023 (0x7fb48570d023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4366694Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb4857111ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4367416Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb4857081d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4367852Z frame #7: <unknown function> + 0x79d655 (0x7fb496bf8655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4368088Z frame #8: <unknown function> + 0xdc253 (0x7fb44dc43253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4368360Z frame #9: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4368572Z frame #10: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4368577Z 
2025-07-17T11:07:26.4368666Z SIGABRT(6), PID: 1811301, Thread 1816163: 
2025-07-17T11:07:26.4369136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4369344Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4369545Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4369747Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4370114Z frame #4: <unknown function> + 0x5a02023 (0x7fe4d7ee2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4370673Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe4d7ee61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4371254Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe4d7edd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4371635Z frame #7: <unknown function> + 0x79d655 (0x7fe4e93cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4371858Z frame #8: <unknown function> + 0xdc253 (0x7fe4a0418253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4372066Z frame #9: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4372278Z frame #10: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4372282Z 
2025-07-17T11:07:26.4372372Z SIGABRT(6), PID: 1811298, Thread 1816212: 
2025-07-17T11:07:26.4372836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4373045Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4373247Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4373445Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4373810Z frame #4: <unknown function> + 0x5a02023 (0x7fb48570d023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4374364Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb4857111ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4374960Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb4857081d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4375411Z frame #7: <unknown function> + 0x79d655 (0x7fb496bf8655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4375694Z frame #8: <unknown function> + 0xdc253 (0x7fb44dc43253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4375981Z frame #9: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4376275Z frame #10: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4376280Z 
2025-07-17T11:07:26.4376373Z SIGABRT(6), PID: 1811301, Thread 1816164: 
2025-07-17T11:07:26.4376919Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4377126Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4377327Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4377531Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4377900Z frame #4: <unknown function> + 0x5a02023 (0x7fe4d7ee2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4378461Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe4d7ee61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4379046Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe4d7edd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4379422Z frame #7: <unknown function> + 0x79d655 (0x7fe4e93cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4379647Z frame #8: <unknown function> + 0xdc253 (0x7fe4a0418253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4379853Z frame #9: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4380068Z frame #10: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4380072Z 
2025-07-17T11:07:26.4380160Z SIGABRT(6), PID: 1811298, Thread 1816213: 
2025-07-17T11:07:26.4380631Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4380836Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4381037Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4381232Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4381598Z frame #4: <unknown function> + 0x5a02023 (0x7fb48570d023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4382155Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb4857111ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4382742Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb4857081d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4383117Z frame #7: <unknown function> + 0x79d655 (0x7fb496bf8655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4383343Z frame #8: <unknown function> + 0xdc253 (0x7fb44dc43253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4383547Z frame #9: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4383869Z frame #10: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4383873Z 
2025-07-17T11:07:26.4383965Z SIGABRT(6), PID: 1811301, Thread 1816165: 
2025-07-17T11:07:26.4384533Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4384809Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4385020Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4385278Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4385656Z frame #4: <unknown function> + 0x5a02023 (0x7fe4d7ee2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4386220Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe4d7ee61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4386816Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe4d7edd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4387192Z frame #7: <unknown function> + 0x79d655 (0x7fe4e93cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4387419Z frame #8: <unknown function> + 0xdc253 (0x7fe4a0418253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4387628Z frame #9: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4387837Z frame #10: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4387841Z 
2025-07-17T11:07:26.4387930Z SIGABRT(6), PID: 1811298, Thread 1816214: 
2025-07-17T11:07:26.4388402Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4388609Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4388810Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4389012Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4389378Z frame #4: <unknown function> + 0x5a02023 (0x7fb48570d023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4389940Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb4857111ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4390523Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb4857081d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4390906Z frame #7: <unknown function> + 0x79d655 (0x7fb496bf8655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4391131Z frame #8: <unknown function> + 0xdc253 (0x7fb44dc43253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4391339Z frame #9: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4391549Z frame #10: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4391554Z 
2025-07-17T11:07:26.4391646Z SIGABRT(6), PID: 1811295, Thread 1811295: 
2025-07-17T11:07:26.4392124Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4392414Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4392648Z frame #2: <unknown function> + 0x7741f (0x7ff934a4f41f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4392956Z frame #3: <unknown function> + 0x76c8e (0x7ff934a4ec8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4393186Z frame #4: <unknown function> + 0x6ad69 (0x7ff934a42d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4393467Z frame #5: <unknown function> + 0x3f54db (0x7ff9c56a74db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4393740Z frame #6: <unknown function> + 0x3e11d8 (0x7ff9c56931d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4393952Z frame #7: <unknown function> + 0xb2f86 (0x7ff9c5364f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4394158Z frame #8: <unknown function> + 0x8e501 (0x7ff9c5340501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4394536Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7ff9f01477ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.4394922Z frame #10: <unknown function> + 0xd307d9 (0x7ffa0e69a7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4395150Z frame #11: <unknown function> + 0x139902 (0x5556c2c70902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4395405Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5556c2c67a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4395634Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4395875Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5556c2c67a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4396100Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4396340Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4396563Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4396800Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4397018Z frame #19: <unknown function> + 0x1474b2 (0x5556c2c7e4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4397220Z frame #20: PyObject_Call + 0xbc (0x5556c2c7ee4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4397441Z frame #21: <unknown function> + 0x1ff69d (0x5556c2d3669d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4397666Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5556c2c6c023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4397909Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5556c2c67cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4398130Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4398364Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5556c2c633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4398586Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4398826Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5556c2c633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4399044Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4399284Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5556c2c67a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4399541Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5556c2c6b4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4399760Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5556c2c7cab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4399972Z frame #32: <unknown function> + 0x205949 (0x5556c2d3c949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4400291Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5556c2c6c023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4400530Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5556c2c67cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4400814Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4401053Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5556c2c633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4401456Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4401754Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5556c2c633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4401977Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4402209Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5556c2c633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4402436Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4402675Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5556c2c643f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4402899Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4403286Z frame #44: <unknown function> + 0x7ab3f7 (0x7ffa0e1153f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4403505Z frame #45: <unknown function> + 0x13bc88 (0x5556c2c72c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4403710Z frame #46: PyObject_Call + 0x20f (0x5556c2c7ef9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4403963Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5556c2c6898d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4404181Z frame #48: <unknown function> + 0x147608 (0x5556c2c7e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4404425Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5556c2c65c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4404650Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4404886Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5556c2c633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4405107Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4405345Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5556c2c643f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4405566Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4405800Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5556c2c633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4406018Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4406258Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5556c2c65c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4406478Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4406718Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5556c2c65c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4406936Z frame #60: <unknown function> + 0x147608 (0x5556c2c7e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4407174Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5556c2c65c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4407388Z frame #62: <unknown function> + 0x147608 (0x5556c2c7e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4407393Z 
2025-07-17T11:07:26.4407487Z SIGABRT(6), PID: 1811301, Thread 1816166: 
2025-07-17T11:07:26.4407976Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4408270Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4408537Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4408745Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4409185Z frame #4: <unknown function> + 0x5a02023 (0x7fe4d7ee2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4409756Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe4d7ee61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4410411Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe4d7edd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4410799Z frame #7: <unknown function> + 0x79d655 (0x7fe4e93cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4411026Z frame #8: <unknown function> + 0xdc253 (0x7fe4a0418253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4411236Z frame #9: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4411451Z frame #10: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4411458Z 
2025-07-17T11:07:26.4411550Z SIGABRT(6), PID: 1811295, Thread 1811379: 
2025-07-17T11:07:26.4412026Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4412234Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4412438Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4412642Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4413153Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4413361Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4413577Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4413582Z 
2025-07-17T11:07:26.4413669Z SIGABRT(6), PID: 1811295, Thread 1811383: 
2025-07-17T11:07:26.4414142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4414346Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4414550Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4414747Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4415248Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4415455Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4415665Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4415672Z 
2025-07-17T11:07:26.4415758Z SIGABRT(6), PID: 1811298, Thread 1816215: 
2025-07-17T11:07:26.4416296Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fb44f9d6b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4416607Z frame #1: <unknown function> + 0x42520 (0x7fb54128e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4416813Z frame #2: <unknown function> + 0x91117 (0x7fb5412dd117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4417084Z frame #3: pthread_cond_wait + 0x211 (0x7fb5412dfa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4417462Z frame #4: <unknown function> + 0x5a02023 (0x7fb48570d023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4418096Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fb4857111ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4418765Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fb4857081d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4419146Z frame #7: <unknown function> + 0x79d655 (0x7fb496bf8655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4419380Z frame #8: <unknown function> + 0xdc253 (0x7fb44dc43253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4419582Z frame #9: <unknown function> + 0x94ac3 (0x7fb5412e0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4419794Z frame #10: <unknown function> + 0x126850 (0x7fb541372850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4419800Z 
2025-07-17T11:07:26.4419887Z SIGABRT(6), PID: 1811295, Thread 1811386: 
2025-07-17T11:07:26.4420363Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4420571Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4420775Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4420973Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4421476Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4421682Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4421893Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4421898Z 
2025-07-17T11:07:26.4421989Z SIGABRT(6), PID: 1811295, Thread 1811389: 
2025-07-17T11:07:26.4422468Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4422675Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4422877Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4423076Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4423586Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4423796Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4424002Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4424008Z 
2025-07-17T11:07:26.4424097Z SIGABRT(6), PID: 1811295, Thread 1811391: 
2025-07-17T11:07:26.4424567Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4424783Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4425086Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4425348Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4425849Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4426120Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4426387Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4426391Z 
2025-07-17T11:07:26.4426481Z SIGABRT(6), PID: 1811301, Thread 1816167: 
2025-07-17T11:07:26.4426953Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe4a21abb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4427162Z frame #1: <unknown function> + 0x42520 (0x7fe593a66520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4427367Z frame #2: <unknown function> + 0x91117 (0x7fe593ab5117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4427565Z frame #3: pthread_cond_wait + 0x211 (0x7fe593ab7a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4427938Z frame #4: <unknown function> + 0x5a02023 (0x7fe4d7ee2023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4428505Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe4d7ee61ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4429099Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe4d7edd1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4429482Z frame #7: <unknown function> + 0x79d655 (0x7fe4e93cd655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4429710Z frame #8: <unknown function> + 0xdc253 (0x7fe4a0418253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4429920Z frame #9: <unknown function> + 0x94ac3 (0x7fe593ab8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4430137Z frame #10: <unknown function> + 0x126850 (0x7fe593b4a850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4430141Z 
2025-07-17T11:07:26.4430232Z SIGABRT(6), PID: 1811295, Thread 1811394: 
2025-07-17T11:07:26.4430703Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4430911Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4431112Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4431309Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4431805Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4432011Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4432218Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4432226Z 
2025-07-17T11:07:26.4432314Z SIGABRT(6), PID: 1811295, Thread 1811397: 
2025-07-17T11:07:26.4432786Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4432988Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4433278Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4433472Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4434042Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4434313Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4434527Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4434592Z 
2025-07-17T11:07:26.4434682Z SIGABRT(6), PID: 1811295, Thread 1811400: 
2025-07-17T11:07:26.4435150Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4435355Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4435563Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4435759Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4436257Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4436462Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4436673Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4436677Z 
2025-07-17T11:07:26.4436762Z SIGABRT(6), PID: 1811295, Thread 1811403: 
2025-07-17T11:07:26.4437230Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4437435Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4437641Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4437835Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4438329Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4438534Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4438742Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4438746Z 
2025-07-17T11:07:26.4438832Z SIGABRT(6), PID: 1811295, Thread 1811405: 
2025-07-17T11:07:26.4439303Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4439507Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4439715Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4439912Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4440409Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4440618Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4440826Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4440830Z 
2025-07-17T11:07:26.4440917Z SIGABRT(6), PID: 1811295, Thread 1811407: 
2025-07-17T11:07:26.4441458Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4441720Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4441929Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4442181Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4442680Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4442950Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4443156Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4443162Z 
2025-07-17T11:07:26.4443251Z SIGABRT(6), PID: 1811295, Thread 1811409: 
2025-07-17T11:07:26.4443718Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4443925Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4444129Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4444328Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4444821Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4445030Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4445240Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4445246Z 
2025-07-17T11:07:26.4445336Z SIGABRT(6), PID: 1811295, Thread 1811411: 
2025-07-17T11:07:26.4445802Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4446007Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4446210Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4446407Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4446902Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4447114Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4447322Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4447326Z 
2025-07-17T11:07:26.4447415Z SIGABRT(6), PID: 1811295, Thread 1811413: 
2025-07-17T11:07:26.4447883Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4448091Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4448295Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4448495Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4448984Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4449191Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4449475Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4449479Z 
2025-07-17T11:07:26.4449631Z SIGABRT(6), PID: 1811295, Thread 1811415: 
2025-07-17T11:07:26.4450109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4450381Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4450670Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4450871Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4451369Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4451580Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4451789Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4451793Z 
2025-07-17T11:07:26.4451885Z SIGABRT(6), PID: 1811295, Thread 1811417: 
2025-07-17T11:07:26.4452355Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4452563Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4452769Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4452964Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4453462Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4453670Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4453881Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4453885Z 
2025-07-17T11:07:26.4453973Z SIGABRT(6), PID: 1811295, Thread 1811419: 
2025-07-17T11:07:26.4454441Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4454645Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4454851Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4455044Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4455540Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4455747Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4456069Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4456074Z 
2025-07-17T11:07:26.4456166Z SIGABRT(6), PID: 1811295, Thread 1811421: 
2025-07-17T11:07:26.4456636Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4456841Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4457044Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4457242Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4457899Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4458111Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4458320Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4458397Z 
2025-07-17T11:07:26.4458487Z SIGABRT(6), PID: 1811295, Thread 1811423: 
2025-07-17T11:07:26.4459030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4459234Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4459438Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4459635Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4460133Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4460338Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4460553Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4460557Z 
2025-07-17T11:07:26.4460644Z SIGABRT(6), PID: 1811295, Thread 1811425: 
2025-07-17T11:07:26.4461108Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4461312Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4461518Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4461710Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4462208Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4462414Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4462624Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4462629Z 
2025-07-17T11:07:26.4462716Z SIGABRT(6), PID: 1811295, Thread 1811428: 
2025-07-17T11:07:26.4463184Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4463393Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4463595Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4463793Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4464289Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4464499Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4464706Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4464709Z 
2025-07-17T11:07:26.4464799Z SIGABRT(6), PID: 1811295, Thread 1811430: 
2025-07-17T11:07:26.4465263Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4465545Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4465748Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4466005Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4466561Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4466771Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4467042Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4467046Z 
2025-07-17T11:07:26.4467136Z SIGABRT(6), PID: 1811295, Thread 1811432: 
2025-07-17T11:07:26.4467604Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4467812Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4468012Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4468210Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4468702Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4468911Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4469118Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4469122Z 
2025-07-17T11:07:26.4469210Z SIGABRT(6), PID: 1811295, Thread 1811434: 
2025-07-17T11:07:26.4469673Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4469881Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4470081Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4470281Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4470768Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4470980Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4471186Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4471190Z 
2025-07-17T11:07:26.4471281Z SIGABRT(6), PID: 1811295, Thread 1811436: 
2025-07-17T11:07:26.4471748Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4471954Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4472157Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4472354Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4472844Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4473052Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4473256Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4473333Z 
2025-07-17T11:07:26.4473423Z SIGABRT(6), PID: 1811295, Thread 1811438: 
2025-07-17T11:07:26.4473952Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4474161Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4474425Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4474623Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4475200Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4475406Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4475622Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4475626Z 
2025-07-17T11:07:26.4475713Z SIGABRT(6), PID: 1811295, Thread 1811440: 
2025-07-17T11:07:26.4476184Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4476390Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4476595Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4476790Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4477285Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4477489Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4477698Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4477702Z 
2025-07-17T11:07:26.4477789Z SIGABRT(6), PID: 1811295, Thread 1811442: 
2025-07-17T11:07:26.4478259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4478467Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4478674Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4478868Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4479362Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4479569Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4479775Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4479780Z 
2025-07-17T11:07:26.4479867Z SIGABRT(6), PID: 1811295, Thread 1811444: 
2025-07-17T11:07:26.4480341Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4480543Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4480752Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4480945Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4481437Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4481722Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4481996Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4482088Z SIGABRT(6), PID: 1811299, Thread 1811299: 
2025-07-17T11:07:26.4482634Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4482901Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4483136Z frame #2: <unknown function> + 0x7741f (0x7f6c9bb7a41f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4483363Z frame #3: <unknown function> + 0x76c8e (0x7f6c9bb79c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4483590Z frame #4: <unknown function> + 0x6ad69 (0x7f6c9bb6dd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4483806Z frame #5: <unknown function> + 0x3f54db (0x7f6d2c7d24db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4484020Z frame #6: <unknown function> + 0x3e11d8 (0x7f6d2c7be1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4484228Z frame #7: <unknown function> + 0xb2f86 (0x7f6d2c48ff86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4484434Z frame #8: <unknown function> + 0x8e501 (0x7f6d2c46b501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4484813Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f6d572727ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.4485201Z frame #10: <unknown function> + 0xd307d9 (0x7f6d757c57d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4485426Z frame #11: <unknown function> + 0x139902 (0x55560df2a902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4485677Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55560df21a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4485905Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4486150Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55560df21a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4486372Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4486616Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4486838Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4487075Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4487297Z frame #19: <unknown function> + 0x1474b2 (0x55560df384b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4487502Z frame #20: PyObject_Call + 0xbc (0x55560df38e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4487720Z frame #21: <unknown function> + 0x1ff69d (0x55560dff069d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4487946Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55560df26023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4488186Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55560df21cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4488415Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4488650Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55560df1d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4488873Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4489105Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55560df1d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4489326Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4489639Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55560df21a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4489955Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55560df254b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4490176Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55560df36ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4490453Z frame #32: <unknown function> + 0x205949 (0x55560dff6949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4490675Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55560df26023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4490977Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55560df21cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4491199Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4491432Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55560df1d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4491656Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4491889Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55560df1d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4492108Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4492344Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55560df1d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4492563Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4492808Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55560df1e3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4493027Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4493410Z frame #44: <unknown function> + 0x7ab3f7 (0x7f6d752403f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4493632Z frame #45: <unknown function> + 0x13bc88 (0x55560df2cc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4493837Z frame #46: PyObject_Call + 0x20f (0x55560df38f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4494076Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55560df2298d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4494295Z frame #48: <unknown function> + 0x147608 (0x55560df38608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4494534Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55560df1fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4494757Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4494991Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55560df1d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4495218Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4495454Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55560df1e3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4495681Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4495979Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55560df1d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4496206Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4496442Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55560df1fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4496664Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4496899Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55560df1fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4497116Z frame #60: <unknown function> + 0x147608 (0x55560df38608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4497446Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55560df1fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4497738Z frame #62: <unknown function> + 0x147608 (0x55560df38608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4497743Z 
2025-07-17T11:07:26.4497746Z 
2025-07-17T11:07:26.4497839Z SIGABRT(6), PID: 1811295, Thread 1811446: 
2025-07-17T11:07:26.4498388Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4498670Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4498879Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4499079Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4499586Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4499794Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4500008Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4500012Z 
2025-07-17T11:07:26.4500104Z SIGABRT(6), PID: 1811299, Thread 1811542: 
2025-07-17T11:07:26.4500572Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4500779Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4500980Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4501176Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4501674Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4501882Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4502090Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4502096Z 
2025-07-17T11:07:26.4502188Z SIGABRT(6), PID: 1811295, Thread 1811448: 
2025-07-17T11:07:26.4502655Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4502866Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4503066Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4503267Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4503762Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4503970Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4504178Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4504182Z 
2025-07-17T11:07:26.4504272Z SIGABRT(6), PID: 1811299, Thread 1811545: 
2025-07-17T11:07:26.4504735Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4504944Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4505220Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4505417Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4505965Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4506236Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4506447Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4506511Z 
2025-07-17T11:07:26.4506607Z SIGABRT(6), PID: 1811295, Thread 1811451: 
2025-07-17T11:07:26.4507074Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4507281Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4507485Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4507680Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4508170Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4508379Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4508585Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4508589Z 
2025-07-17T11:07:26.4508680Z SIGABRT(6), PID: 1811299, Thread 1811548: 
2025-07-17T11:07:26.4509139Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4509345Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4509544Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4509747Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4510239Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4510444Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4510654Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4510658Z 
2025-07-17T11:07:26.4510748Z SIGABRT(6), PID: 1811295, Thread 1811453: 
2025-07-17T11:07:26.4511216Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4511425Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4511630Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4511824Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4512320Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4512526Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4512732Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4512736Z 
2025-07-17T11:07:26.4512822Z SIGABRT(6), PID: 1811299, Thread 1811551: 
2025-07-17T11:07:26.4513361Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4513646Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4513853Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4514106Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4514607Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4514875Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4515087Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4515092Z 
2025-07-17T11:07:26.4515188Z SIGABRT(6), PID: 1811295, Thread 1811455: 
2025-07-17T11:07:26.4515661Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4515866Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4516071Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4516267Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4516760Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4516967Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4517180Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4517186Z 
2025-07-17T11:07:26.4517274Z SIGABRT(6), PID: 1811299, Thread 1811554: 
2025-07-17T11:07:26.4517746Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4517949Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4518154Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4518346Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4518840Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4519043Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4519254Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4519258Z 
2025-07-17T11:07:26.4519347Z SIGABRT(6), PID: 1811295, Thread 1811457: 
2025-07-17T11:07:26.4519818Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4520026Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4520226Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4520420Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4520915Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4521122Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4521403Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4521408Z 
2025-07-17T11:07:26.4521559Z SIGABRT(6), PID: 1811299, Thread 1811557: 
2025-07-17T11:07:26.4522036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4522301Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4522561Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4522757Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4523257Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4523465Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4523670Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4523675Z 
2025-07-17T11:07:26.4523764Z SIGABRT(6), PID: 1811295, Thread 1811459: 
2025-07-17T11:07:26.4524232Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4524440Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4524643Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4524843Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4525337Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4525548Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4525754Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4525758Z 
2025-07-17T11:07:26.4525848Z SIGABRT(6), PID: 1811299, Thread 1811560: 
2025-07-17T11:07:26.4526313Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4526522Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4526720Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4526916Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4527410Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4527618Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4527826Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4527829Z 
2025-07-17T11:07:26.4527921Z SIGABRT(6), PID: 1811295, Thread 1811461: 
2025-07-17T11:07:26.4528391Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4528601Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4528800Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4528997Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4529565Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4529955Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4530167Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4530171Z 
2025-07-17T11:07:26.4530324Z SIGABRT(6), PID: 1811299, Thread 1811563: 
2025-07-17T11:07:26.4530851Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4531057Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4531261Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4531458Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4531954Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4532158Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4532368Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4532372Z 
2025-07-17T11:07:26.4532459Z SIGABRT(6), PID: 1811295, Thread 1811463: 
2025-07-17T11:07:26.4532928Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4533131Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4533334Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4533529Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4534026Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4534232Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4534444Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4534449Z 
2025-07-17T11:07:26.4534537Z SIGABRT(6), PID: 1811299, Thread 1811566: 
2025-07-17T11:07:26.4535008Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4535211Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4535415Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4535610Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4536172Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4536379Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4536590Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4536594Z 
2025-07-17T11:07:26.4536680Z SIGABRT(6), PID: 1811295, Thread 1811465: 
2025-07-17T11:07:26.4537150Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4537446Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4537652Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4537920Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4538490Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4538700Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4538988Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4538992Z 
2025-07-17T11:07:26.4539078Z SIGABRT(6), PID: 1811299, Thread 1811569: 
2025-07-17T11:07:26.4539545Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4539750Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4539955Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4540146Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4540644Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4540851Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4541063Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4541067Z 
2025-07-17T11:07:26.4541154Z SIGABRT(6), PID: 1811295, Thread 1811467: 
2025-07-17T11:07:26.4541625Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4541834Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4542035Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4542235Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4542729Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4542938Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4543144Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4543148Z 
2025-07-17T11:07:26.4543237Z SIGABRT(6), PID: 1811299, Thread 1811572: 
2025-07-17T11:07:26.4543705Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4543912Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4544113Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4544313Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4544805Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4545013Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4545218Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4545313Z 
2025-07-17T11:07:26.4545406Z SIGABRT(6), PID: 1811295, Thread 1811469: 
2025-07-17T11:07:26.4545935Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4546148Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4546415Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4546620Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4547176Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4547386Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4547591Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4547598Z 
2025-07-17T11:07:26.4547686Z SIGABRT(6), PID: 1811299, Thread 1811575: 
2025-07-17T11:07:26.4548153Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4548359Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4548559Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4548758Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4549248Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4549455Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4549662Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4549666Z 
2025-07-17T11:07:26.4549758Z SIGABRT(6), PID: 1811295, Thread 1811472: 
2025-07-17T11:07:26.4550223Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4550432Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4550634Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4550836Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4551329Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4551540Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4551746Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4551754Z 
2025-07-17T11:07:26.4551841Z SIGABRT(6), PID: 1811299, Thread 1811578: 
2025-07-17T11:07:26.4552308Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4552513Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4552716Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4552909Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4553403Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4553684Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4553954Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4553958Z 
2025-07-17T11:07:26.4554049Z SIGABRT(6), PID: 1811295, Thread 1811473: 
2025-07-17T11:07:26.4554605Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4554871Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4555077Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4555271Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4555770Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4555979Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4556188Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4556277Z SIGABRT(6), PID: 1811299, Thread 1811581: 
2025-07-17T11:07:26.4556743Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4556947Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4557149Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4557343Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4557839Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4558043Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4558250Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4558254Z 
2025-07-17T11:07:26.4558257Z 
2025-07-17T11:07:26.4558344Z SIGABRT(6), PID: 1811295, Thread 1811474: 
2025-07-17T11:07:26.4558811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4559016Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4559220Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4559415Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4559913Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4560118Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4560327Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4560331Z 
2025-07-17T11:07:26.4560417Z SIGABRT(6), PID: 1811299, Thread 1811584: 
2025-07-17T11:07:26.4560889Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4561095Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4561299Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4561573Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4562130Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4562339Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4562610Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4562671Z 
2025-07-17T11:07:26.4562763Z SIGABRT(6), PID: 1811295, Thread 1811475: 
2025-07-17T11:07:26.4563234Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4563440Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4563643Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4563838Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4564329Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4564540Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4564746Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4564751Z 
2025-07-17T11:07:26.4564839Z SIGABRT(6), PID: 1811299, Thread 1811588: 
2025-07-17T11:07:26.4565313Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4565521Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4565717Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4565915Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4566409Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4566616Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4566823Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4566827Z 
2025-07-17T11:07:26.4566916Z SIGABRT(6), PID: 1811299, Thread 1811591: 
2025-07-17T11:07:26.4567377Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4567582Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4567784Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4567982Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4568474Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4568680Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4568885Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4568975Z SIGABRT(6), PID: 1811295, Thread 1811476: 
2025-07-17T11:07:26.4569440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4569718Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4569984Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4570186Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4570739Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4571032Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4571239Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4571243Z 
2025-07-17T11:07:26.4571246Z 
2025-07-17T11:07:26.4571335Z SIGABRT(6), PID: 1811299, Thread 1811594: 
2025-07-17T11:07:26.4571805Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4572014Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4572217Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4572418Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4572917Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4573124Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4573328Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4573337Z 
2025-07-17T11:07:26.4573423Z SIGABRT(6), PID: 1811295, Thread 1811477: 
2025-07-17T11:07:26.4573889Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4574094Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4574299Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4574496Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4574991Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4575197Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4575405Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4575411Z 
2025-07-17T11:07:26.4575497Z SIGABRT(6), PID: 1811299, Thread 1811597: 
2025-07-17T11:07:26.4576042Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4576246Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4576451Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4576646Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4577141Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4577347Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4577654Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4577658Z 
2025-07-17T11:07:26.4577746Z SIGABRT(6), PID: 1811295, Thread 1811478: 
2025-07-17T11:07:26.4578289Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4578568Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4578776Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4579042Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4579538Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4579745Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4579956Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4579960Z 
2025-07-17T11:07:26.4580049Z SIGABRT(6), PID: 1811297, Thread 1811297: 
2025-07-17T11:07:26.4580522Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4580729Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4580960Z frame #2: <unknown function> + 0x77435 (0x7f6f88c06435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4581188Z frame #3: <unknown function> + 0x76c8e (0x7f6f88c05c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4581416Z frame #4: <unknown function> + 0x6ad69 (0x7f6f88bf9d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4581627Z frame #5: <unknown function> + 0x3f54db (0x7f701985e4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4581834Z frame #6: <unknown function> + 0x3e11d8 (0x7f701984a1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4582041Z frame #7: <unknown function> + 0xb2f86 (0x7f701951bf86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4582248Z frame #8: <unknown function> + 0x8e501 (0x7f70194f7501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4582627Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f70442fe7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.4583012Z frame #10: <unknown function> + 0xd307d9 (0x7f70628517d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4583241Z frame #11: <unknown function> + 0x139902 (0x55b2cbf15902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4583492Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55b2cbf0ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4583723Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4583971Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55b2cbf0ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4584195Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4584438Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4584659Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4584900Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4585117Z frame #19: <unknown function> + 0x1474b2 (0x55b2cbf234b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4585321Z frame #20: PyObject_Call + 0xbc (0x55b2cbf23e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4585618Z frame #21: <unknown function> + 0x1ff69d (0x55b2cbfdb69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4585844Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55b2cbf11023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4586143Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55b2cbf0ccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4586373Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4586664Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55b2cbf083bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4586948Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4587181Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55b2cbf083bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4587403Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4587642Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55b2cbf0ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4587897Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55b2cbf104b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4588115Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55b2cbf21ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4588335Z frame #32: <unknown function> + 0x205949 (0x55b2cbfe1949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4588557Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55b2cbf11023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4588801Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55b2cbf0ccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4589021Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4589259Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55b2cbf083bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4589479Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4589714Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55b2cbf083bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4589937Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4590172Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55b2cbf083bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4590395Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4590635Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55b2cbf093f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4590857Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4591246Z frame #44: <unknown function> + 0x7ab3f7 (0x7f70622cc3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4591466Z frame #45: <unknown function> + 0x13bc88 (0x55b2cbf17c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4591673Z frame #46: PyObject_Call + 0x20f (0x55b2cbf23f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4591914Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55b2cbf0d98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4592131Z frame #48: <unknown function> + 0x147608 (0x55b2cbf23608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4592371Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b2cbf0ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4592599Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4592833Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55b2cbf083bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4593054Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4593365Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55b2cbf093f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4593586Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4593884Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55b2cbf083bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4594112Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4594408Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b2cbf0ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4594688Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4594932Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b2cbf0ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4595162Z frame #60: <unknown function> + 0x147608 (0x55b2cbf23608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4595402Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b2cbf0ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4595619Z frame #62: <unknown function> + 0x147608 (0x55b2cbf23608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4595624Z 
2025-07-17T11:07:26.4595715Z SIGABRT(6), PID: 1811299, Thread 1811600: 
2025-07-17T11:07:26.4596201Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4596413Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4596623Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4596821Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4597323Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4597530Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4597743Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4597747Z 
2025-07-17T11:07:26.4597836Z SIGABRT(6), PID: 1811295, Thread 1811479: 
2025-07-17T11:07:26.4598314Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4598525Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4598725Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4598924Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4599426Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4599638Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4599847Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4599851Z 
2025-07-17T11:07:26.4599941Z SIGABRT(6), PID: 1811297, Thread 1811502: 
2025-07-17T11:07:26.4600410Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4600616Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4600815Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4601011Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4601582Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4601849Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4602061Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4602231Z SIGABRT(6), PID: 1811299, Thread 1811603: 
2025-07-17T11:07:26.4602699Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4602969Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4603173Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4603372Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4603867Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4604074Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4604280Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4604285Z 
2025-07-17T11:07:26.4604288Z 
2025-07-17T11:07:26.4604377Z SIGABRT(6), PID: 1811295, Thread 1811480: 
2025-07-17T11:07:26.4604843Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4605050Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4605256Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4605457Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4605950Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4606158Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4606364Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4606369Z 
2025-07-17T11:07:26.4606459Z SIGABRT(6), PID: 1811295, Thread 1811481: 
2025-07-17T11:07:26.4606923Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4607129Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4607332Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4607528Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4608026Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4608236Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4608440Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4608448Z 
2025-07-17T11:07:26.4608534Z SIGABRT(6), PID: 1811297, Thread 1811503: 
2025-07-17T11:07:26.4609001Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4609281Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4609484Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4609741Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4610298Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4610506Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4610776Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4610863Z SIGABRT(6), PID: 1811299, Thread 1811606: 
2025-07-17T11:07:26.4611352Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4611562Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4611768Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4611962Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4612458Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4612663Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4612873Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4612877Z 
2025-07-17T11:07:26.4612881Z 
2025-07-17T11:07:26.4612968Z SIGABRT(6), PID: 1811295, Thread 1811482: 
2025-07-17T11:07:26.4613437Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4613648Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4613858Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4614054Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4614557Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4614762Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4614968Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4614972Z 
2025-07-17T11:07:26.4615057Z SIGABRT(6), PID: 1811295, Thread 1811483: 
2025-07-17T11:07:26.4615537Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4615741Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4616028Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4616227Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4616724Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4616930Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4617138Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4617226Z 
2025-07-17T11:07:26.4617317Z SIGABRT(6), PID: 1811297, Thread 1811504: 
2025-07-17T11:07:26.4617854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4618061Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4618339Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4618538Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4619110Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4619315Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4619525Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4619615Z SIGABRT(6), PID: 1811299, Thread 1811609: 
2025-07-17T11:07:26.4620085Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4620289Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4620489Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4620684Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4621181Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4621387Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4621599Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4621603Z 
2025-07-17T11:07:26.4621606Z 
2025-07-17T11:07:26.4621698Z SIGABRT(6), PID: 1811295, Thread 1811484: 
2025-07-17T11:07:26.4622169Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4622377Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4622579Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4622776Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4623271Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4623480Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4623688Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4623692Z 
2025-07-17T11:07:26.4623783Z SIGABRT(6), PID: 1811299, Thread 1811612: 
2025-07-17T11:07:26.4624246Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4624452Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4624656Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4624851Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4625351Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4625633Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4625905Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4625909Z 
2025-07-17T11:07:26.4626001Z SIGABRT(6), PID: 1811297, Thread 1811505: 
2025-07-17T11:07:26.4626524Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4626792Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4626991Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4627186Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4627677Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4627888Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4628095Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4628099Z 
2025-07-17T11:07:26.4628191Z SIGABRT(6), PID: 1811295, Thread 1811485: 
2025-07-17T11:07:26.4628661Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4628868Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4629067Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4629264Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4629760Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4629970Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4630178Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4630270Z SIGABRT(6), PID: 1811299, Thread 1811615: 
2025-07-17T11:07:26.4630737Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4630942Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4631144Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4631338Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4631833Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4632038Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4632247Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4632253Z 
2025-07-17T11:07:26.4632339Z SIGABRT(6), PID: 1811297, Thread 1811506: 
2025-07-17T11:07:26.4632809Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4633010Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4633214Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4633482Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4634075Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4634284Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4634549Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4634553Z 
2025-07-17T11:07:26.4634614Z 
2025-07-17T11:07:26.4634706Z SIGABRT(6), PID: 1811295, Thread 1811486: 
2025-07-17T11:07:26.4635185Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4635390Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4635597Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4635796Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4636290Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4636499Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4636707Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4636713Z 
2025-07-17T11:07:26.4636799Z SIGABRT(6), PID: 1811299, Thread 1811618: 
2025-07-17T11:07:26.4637270Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4637472Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4637676Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4637871Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4638369Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4638574Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4638785Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4638789Z 
2025-07-17T11:07:26.4638876Z SIGABRT(6), PID: 1811297, Thread 1811507: 
2025-07-17T11:07:26.4639340Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4639543Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4639746Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4639946Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4640446Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4640652Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4640861Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4640866Z 
2025-07-17T11:07:26.4640956Z SIGABRT(6), PID: 1811299, Thread 1811621: 
2025-07-17T11:07:26.4641422Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4641701Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4641960Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4642160Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4642714Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4642983Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4643190Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4643194Z 
2025-07-17T11:07:26.4643284Z SIGABRT(6), PID: 1811295, Thread 1811487: 
2025-07-17T11:07:26.4643751Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4643959Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4644160Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4644362Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4644852Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4645063Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4645268Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4645274Z 
2025-07-17T11:07:26.4645363Z SIGABRT(6), PID: 1811297, Thread 1811508: 
2025-07-17T11:07:26.4645827Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4646035Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4646235Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4646435Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4646926Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4647135Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4647338Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4647343Z 
2025-07-17T11:07:26.4647435Z SIGABRT(6), PID: 1811299, Thread 1811624: 
2025-07-17T11:07:26.4647901Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4648105Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4648306Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4648502Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4648994Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4649204Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4649489Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4649493Z 
2025-07-17T11:07:26.4649584Z SIGABRT(6), PID: 1811295, Thread 1811488: 
2025-07-17T11:07:26.4650112Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4650381Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4650586Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4650845Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4651337Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4651547Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4651756Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4651763Z 
2025-07-17T11:07:26.4651851Z SIGABRT(6), PID: 1811297, Thread 1811509: 
2025-07-17T11:07:26.4652323Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4652527Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4652731Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4652925Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4653424Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4653631Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4653840Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4653844Z 
2025-07-17T11:07:26.4653932Z SIGABRT(6), PID: 1811299, Thread 1811627: 
2025-07-17T11:07:26.4654401Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4654605Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4654811Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4655005Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4655507Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4655713Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4655988Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4655993Z 
2025-07-17T11:07:26.4656081Z SIGABRT(6), PID: 1811295, Thread 1811489: 
2025-07-17T11:07:26.4656554Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4656761Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4656966Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4657163Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4657866Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4658142Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4658354Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4658358Z 
2025-07-17T11:07:26.4658518Z SIGABRT(6), PID: 1811297, Thread 1811510: 
2025-07-17T11:07:26.4658991Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4659271Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4659475Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4659669Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4660166Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4660369Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4660575Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4660581Z 
2025-07-17T11:07:26.4660667Z SIGABRT(6), PID: 1811299, Thread 1811629: 
2025-07-17T11:07:26.4661136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4661339Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4661539Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4661736Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4662230Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4662439Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4662648Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4662652Z 
2025-07-17T11:07:26.4662744Z SIGABRT(6), PID: 1811295, Thread 1811490: 
2025-07-17T11:07:26.4663207Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4663412Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4663615Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4663810Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4664304Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4664512Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4664717Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4664723Z 
2025-07-17T11:07:26.4664812Z SIGABRT(6), PID: 1811297, Thread 1811511: 
2025-07-17T11:07:26.4665274Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4665481Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4665775Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4666031Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4666530Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4666796Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4667062Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4667153Z SIGABRT(6), PID: 1811299, Thread 1811634: 
2025-07-17T11:07:26.4667618Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4667825Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4668026Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4668224Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4668720Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4668929Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4669136Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4669140Z 
2025-07-17T11:07:26.4669143Z 
2025-07-17T11:07:26.4669233Z SIGABRT(6), PID: 1811295, Thread 1811491: 
2025-07-17T11:07:26.4669695Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4669905Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4670108Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4670307Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4670801Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4671010Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4671218Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4671222Z 
2025-07-17T11:07:26.4671310Z SIGABRT(6), PID: 1811299, Thread 1811636: 
2025-07-17T11:07:26.4671778Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4671986Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4672186Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4672380Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4672874Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4673080Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4673288Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4673292Z 
2025-07-17T11:07:26.4673453Z SIGABRT(6), PID: 1811297, Thread 1811512: 
2025-07-17T11:07:26.4673920Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4674181Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4674385Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4674637Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4675190Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4675396Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4675601Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4675607Z 
2025-07-17T11:07:26.4675692Z SIGABRT(6), PID: 1811295, Thread 1811492: 
2025-07-17T11:07:26.4676164Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4676368Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4676573Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4676767Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4677262Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4677466Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4677677Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4677681Z 
2025-07-17T11:07:26.4677766Z SIGABRT(6), PID: 1811299, Thread 1811639: 
2025-07-17T11:07:26.4678237Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4678443Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4678649Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4678844Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4679337Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4679542Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4679754Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4679758Z 
2025-07-17T11:07:26.4679848Z SIGABRT(6), PID: 1811295, Thread 1811493: 
2025-07-17T11:07:26.4680324Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4680530Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4680732Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4680926Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4681422Z frame #4: <unknown function> + 0x3475ab (0x7ffab568c5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4681704Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4681913Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4682061Z SIGABRT(6), PID: 1811297, Thread 1811513: 
2025-07-17T11:07:26.4682542Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4682809Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4683076Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4683271Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4683768Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4683973Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4684180Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4684183Z 
2025-07-17T11:07:26.4684187Z 
2025-07-17T11:07:26.4684275Z SIGABRT(6), PID: 1811299, Thread 1811643: 
2025-07-17T11:07:26.4684741Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4684949Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4685149Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4685344Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4685837Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4686046Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4686254Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4686257Z 
2025-07-17T11:07:26.4686347Z SIGABRT(6), PID: 1811295, Thread 1812350: 
2025-07-17T11:07:26.4686814Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4687269Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7ff9c6ee608a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4687476Z frame #2: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4687672Z frame #3: pthread_kill + 0x12c (0x7ffab87f69fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4687834Z frame #4: raise + 0x16 (0x7ffab87a2476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4687994Z frame #5: abort + 0xd3 (0x7ffab87887f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4688229Z frame #6: <unknown function> + 0x972cc (0x7ff934a6f2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4688459Z frame #7: <unknown function> + 0x95f47 (0x7ff934a6df47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4688683Z frame #8: <unknown function> + 0x34b71 (0x7ff934a0cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4688893Z frame #9: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4689103Z frame #10: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4689107Z 
2025-07-17T11:07:26.4689197Z SIGABRT(6), PID: 1811297, Thread 1811514: 
2025-07-17T11:07:26.4689740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4690008Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4690216Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4690475Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4690975Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4691263Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4691469Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4691475Z 
2025-07-17T11:07:26.4691567Z SIGABRT(6), PID: 1811299, Thread 1811646: 
2025-07-17T11:07:26.4692031Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4692241Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4692440Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4692637Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4693131Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4693339Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4693545Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4693638Z SIGABRT(6), PID: 1811295, Thread 1815684: 
2025-07-17T11:07:26.4694107Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4694313Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4694480Z frame #2: __poll + 0x4f (0x7ffab8878bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4694887Z frame #3: <unknown function> + 0x489f (0x7ffab838489f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.4695115Z frame #4: <unknown function> + 0x1483bc (0x5556c2c7f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4695368Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4695589Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4701522Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4701804Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4702073Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5556c2c67a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4702312Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4702574Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5556c2c65c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4702808Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4703049Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4703272Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4703658Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4703959Z frame #16: <unknown function> + 0x147734 (0x5556c2c7e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4704186Z frame #17: <unknown function> + 0x21ca82 (0x5556c2d53a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4704481Z frame #18: <unknown function> + 0x21c9e4 (0x5556c2d539e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4704712Z frame #19: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4705005Z frame #20: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4705105Z SIGABRT(6), PID: 1811297, Thread 1811515: 
2025-07-17T11:07:26.4705616Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4705841Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4706047Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4706254Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4706770Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4706988Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4707198Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4707212Z 
2025-07-17T11:07:26.4707216Z 
2025-07-17T11:07:26.4707219Z 
2025-07-17T11:07:26.4707315Z SIGABRT(6), PID: 1811297, Thread 1811516: 
2025-07-17T11:07:26.4707803Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4708010Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4708214Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4708418Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4708929Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4709135Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4709346Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4709353Z 
2025-07-17T11:07:26.4709443Z SIGABRT(6), PID: 1811299, Thread 1811648: 
2025-07-17T11:07:26.4709923Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4710133Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4710335Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4710533Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4711036Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4711241Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4711455Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4711529Z 
2025-07-17T11:07:26.4711620Z SIGABRT(6), PID: 1811297, Thread 1811517: 
2025-07-17T11:07:26.4712150Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4712365Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4712632Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4712834Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4713407Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4713615Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4713821Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4713825Z 
2025-07-17T11:07:26.4713912Z SIGABRT(6), PID: 1811299, Thread 1811650: 
2025-07-17T11:07:26.4714387Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4714595Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4714798Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4714994Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4715497Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4715701Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4715912Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4715916Z 
2025-07-17T11:07:26.4716006Z SIGABRT(6), PID: 1811295, Thread 1815685: 
2025-07-17T11:07:26.4716484Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4716696Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4716899Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4717125Z frame #3: pthread_cond_clockwait + 0x1fd (0x7ffab87f42dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4717573Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7ff9f2c2e048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.4718008Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7ff9f2c30bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.4718244Z frame #6: <unknown function> + 0xdc253 (0x7ff9c5152253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4718454Z frame #7: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4718661Z frame #8: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4718665Z 
2025-07-17T11:07:26.4718761Z SIGABRT(6), PID: 1811297, Thread 1811518: 
2025-07-17T11:07:26.4719233Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4719440Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4719739Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4719941Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4720501Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4720772Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4720981Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4721135Z SIGABRT(6), PID: 1811299, Thread 1811651: 
2025-07-17T11:07:26.4721602Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4721808Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4722013Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4722213Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4722711Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4722919Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4723125Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4723131Z 
2025-07-17T11:07:26.4723134Z 
2025-07-17T11:07:26.4723223Z SIGABRT(6), PID: 1811295, Thread 1815686: 
2025-07-17T11:07:26.4723686Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4723895Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4724096Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4724321Z frame #3: pthread_cond_clockwait + 0x1fd (0x7ffab87f42dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4724792Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7ff9f2c28741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.4725034Z frame #5: <unknown function> + 0xdc253 (0x7ff9c5152253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4725256Z frame #6: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4725471Z frame #7: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4725475Z 
2025-07-17T11:07:26.4725565Z SIGABRT(6), PID: 1811299, Thread 1811654: 
2025-07-17T11:07:26.4726067Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4726280Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4726488Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4726692Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4727205Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4727413Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4727623Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4727700Z 
2025-07-17T11:07:26.4727799Z SIGABRT(6), PID: 1811297, Thread 1811519: 
2025-07-17T11:07:26.4728330Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4728546Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4728752Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4729035Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4729596Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4729803Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4730009Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4730016Z 
2025-07-17T11:07:26.4730108Z SIGABRT(6), PID: 1811295, Thread 1815687: 
2025-07-17T11:07:26.4730575Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4730782Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4730952Z frame #2: ioctl + 0x3f (0x7ffab887a94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4731193Z frame #3: <unknown function> + 0x13eef0 (0x7ff934b16ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4731427Z frame #4: <unknown function> + 0x1378c4 (0x7ff934b0f8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4731661Z frame #5: <unknown function> + 0x95b2b (0x7ff934a6db2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4731882Z frame #6: <unknown function> + 0x34b71 (0x7ff934a0cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4732097Z frame #7: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4732306Z frame #8: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4732310Z 
2025-07-17T11:07:26.4732402Z SIGABRT(6), PID: 1811299, Thread 1811656: 
2025-07-17T11:07:26.4732875Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4733082Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4733290Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4733491Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4733986Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4734195Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4734402Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4734407Z 
2025-07-17T11:07:26.4734496Z SIGABRT(6), PID: 1811297, Thread 1811521: 
2025-07-17T11:07:26.4734964Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4735172Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4735374Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4735573Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4736235Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4736545Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4736765Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4736769Z 
2025-07-17T11:07:26.4736936Z SIGABRT(6), PID: 1811295, Thread 1816104: 
2025-07-17T11:07:26.4737429Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4737718Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4737890Z frame #2: __poll + 0x4f (0x7ffab8878bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4738107Z frame #3: <unknown function> + 0xd4540 (0x7ff97cc86540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.4738315Z frame #4: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4738527Z frame #5: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4738531Z 
2025-07-17T11:07:26.4738623Z SIGABRT(6), PID: 1811299, Thread 1811658: 
2025-07-17T11:07:26.4739101Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4739311Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4739515Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4739715Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4740223Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4740434Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4740643Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4740647Z 
2025-07-17T11:07:26.4740737Z SIGABRT(6), PID: 1811297, Thread 1811523: 
2025-07-17T11:07:26.4741202Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4741407Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4741606Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4741804Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4742300Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4742509Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4742715Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4742719Z 
2025-07-17T11:07:26.4742811Z SIGABRT(6), PID: 1811295, Thread 1816114: 
2025-07-17T11:07:26.4743280Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4743486Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4743652Z frame #2: __poll + 0x4f (0x7ffab8878bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4743868Z frame #3: <unknown function> + 0xd65a2 (0x7ff97cc885a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.4744162Z frame #4: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4744428Z frame #5: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4744433Z 
2025-07-17T11:07:26.4744523Z SIGABRT(6), PID: 1811299, Thread 1811660: 
2025-07-17T11:07:26.4745053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4745320Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4745527Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4745730Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4746236Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4746446Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4746658Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4746662Z 
2025-07-17T11:07:26.4746749Z SIGABRT(6), PID: 1811295, Thread 1816121: 
2025-07-17T11:07:26.4747218Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4747424Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4747632Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4747828Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4748046Z frame #4: <unknown function> + 0xd1e7f (0x7ff97cc83e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.4748251Z frame #5: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4748458Z frame #6: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4748462Z 
2025-07-17T11:07:26.4748548Z SIGABRT(6), PID: 1811297, Thread 1811525: 
2025-07-17T11:07:26.4749018Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4749223Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4749424Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4749620Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4750121Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4750326Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4750532Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4750537Z 
2025-07-17T11:07:26.4750625Z SIGABRT(6), PID: 1811299, Thread 1811662: 
2025-07-17T11:07:26.4751095Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4751302Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4751505Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4751780Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4752345Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4752556Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4752825Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4752830Z 
2025-07-17T11:07:26.4752921Z SIGABRT(6), PID: 1811296, Thread 1811296: 
2025-07-17T11:07:26.4753451Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4753658Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4753889Z frame #2: <unknown function> + 0x77435 (0x7f1cf343c435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4754120Z frame #3: <unknown function> + 0x76c8e (0x7f1cf343bc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4754346Z frame #4: <unknown function> + 0x6ad69 (0x7f1cf342fd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4754561Z frame #5: <unknown function> + 0x3f54db (0x7f1d840944db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4754773Z frame #6: <unknown function> + 0x3e11d8 (0x7f1d840801d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4754984Z frame #7: <unknown function> + 0xb2f86 (0x7f1d83d51f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4755205Z frame #8: <unknown function> + 0x8e501 (0x7f1d83d2d501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4756053Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f1daeb347ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.4756944Z frame #10: <unknown function> + 0xd307d9 (0x7f1dcd0877d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4757657Z frame #11: <unknown function> + 0x139902 (0x560c8d9e2902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4758242Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x560c8d9d9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4758824Z frame #13: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4759452Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x560c8d9d9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4760032Z frame #15: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4760595Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4761161Z frame #17: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4761735Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4762289Z frame #19: <unknown function> + 0x1474b2 (0x560c8d9f04b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4762836Z frame #20: PyObject_Call + 0xbc (0x560c8d9f0e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4763373Z frame #21: <unknown function> + 0x1ff69d (0x560c8daa869d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4763917Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x560c8d9de023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4764504Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x560c8d9d9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4765102Z frame #24: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4765667Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x560c8d9d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4766236Z frame #26: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4767032Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x560c8d9d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4767739Z frame #28: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4768301Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x560c8d9d9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4768965Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x560c8d9dd4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4769542Z frame #31: _PyObject_Call_Prepend + 0x69 (0x560c8d9eeab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4770145Z frame #32: <unknown function> + 0x205949 (0x560c8daae949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4770686Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x560c8d9de023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4771249Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x560c8d9d9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4771819Z frame #35: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4772381Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x560c8d9d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4772936Z frame #37: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4773495Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x560c8d9d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4774055Z frame #39: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4774604Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x560c8d9d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4775162Z frame #41: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4775729Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x560c8d9d63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4776363Z frame #43: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4777073Z frame #44: <unknown function> + 0x7ab3f7 (0x7f1dccb023f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4777773Z frame #45: <unknown function> + 0x13bc88 (0x560c8d9e4c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4778298Z frame #46: PyObject_Call + 0x20f (0x560c8d9f0f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4778845Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x560c8d9da98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4779403Z frame #48: <unknown function> + 0x147608 (0x560c8d9f0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4779956Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x560c8d9d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4780513Z frame #50: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4781069Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x560c8d9d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4781628Z frame #52: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4782187Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x560c8d9d63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4782745Z frame #54: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4783303Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x560c8d9d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4783861Z frame #56: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4784428Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x560c8d9d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4784985Z frame #58: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4785644Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x560c8d9d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4786289Z frame #60: <unknown function> + 0x147608 (0x560c8d9f0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4786871Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x560c8d9d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4787510Z frame #62: <unknown function> + 0x147608 (0x560c8d9f0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4787831Z 
2025-07-17T11:07:26.4788009Z SIGABRT(6), PID: 1811295, Thread 1816133: 
2025-07-17T11:07:26.4788683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4789469Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4789986Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4790496Z frame #3: <unknown function> + 0x9ca53 (0x7ffab87fca53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4791071Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5556c2c4bbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4791659Z frame #5: <unknown function> + 0x1d314f (0x5556c2d0a14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4792202Z frame #6: <unknown function> + 0x1d2f6b (0x5556c2d09f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4792742Z frame #7: <unknown function> + 0x6df36 (0x5556c2ba4f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4793303Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4793868Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4794431Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4794994Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4795555Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4796113Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4796672Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4797225Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4797781Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4798335Z frame #17: <unknown function> + 0x147734 (0x5556c2c7e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4798864Z frame #18: <unknown function> + 0x21ca82 (0x5556c2d53a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4799395Z frame #19: <unknown function> + 0x21c9e4 (0x5556c2d539e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4799929Z frame #20: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4800457Z frame #21: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4800768Z 
2025-07-17T11:07:26.4800867Z SIGABRT(6), PID: 1811297, Thread 1811527: 
2025-07-17T11:07:26.4801536Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4802314Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4802822Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4803328Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4804218Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4805084Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4805619Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4805991Z 
2025-07-17T11:07:26.4806088Z SIGABRT(6), PID: 1811299, Thread 1811664: 
2025-07-17T11:07:26.4806809Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4807574Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4808080Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4808580Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4809380Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4810178Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4810692Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4811004Z 
2025-07-17T11:07:26.4811099Z SIGABRT(6), PID: 1811296, Thread 1811340: 
2025-07-17T11:07:26.4811738Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4812504Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4813011Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4813512Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4814303Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4815100Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4815608Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4815989Z 
2025-07-17T11:07:26.4816080Z SIGABRT(6), PID: 1811299, Thread 1811666: 
2025-07-17T11:07:26.4816728Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4817486Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4817991Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4818488Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4819277Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4820065Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4820579Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4820885Z 
2025-07-17T11:07:26.4820975Z SIGABRT(6), PID: 1811297, Thread 1811528: 
2025-07-17T11:07:26.4821613Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4822470Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4822967Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4823540Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4824406Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4825200Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4825914Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4826219Z 
2025-07-17T11:07:26.4826308Z SIGABRT(6), PID: 1811300, Thread 1811300: 
2025-07-17T11:07:26.4826949Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4827706Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4828244Z frame #2: <unknown function> + 0x77435 (0x7f20a9d8c435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4828803Z frame #3: <unknown function> + 0x76c8e (0x7f20a9d8bc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4829354Z frame #4: <unknown function> + 0x6ad69 (0x7f20a9d7fd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.4829890Z frame #5: <unknown function> + 0x3f54db (0x7f213a9e44db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4830410Z frame #6: <unknown function> + 0x3e11d8 (0x7f213a9d01d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4830924Z frame #7: <unknown function> + 0xb2f86 (0x7f213a6a1f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4831437Z frame #8: <unknown function> + 0x8e501 (0x7f213a67d501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.4832122Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f21654847ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.4832971Z frame #10: <unknown function> + 0xd307d9 (0x7f21839d77d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4833667Z frame #11: <unknown function> + 0x139902 (0x5558ca0e2902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4834242Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5558ca0d9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4834820Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4835389Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5558ca0d9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4835962Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4836528Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4837091Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4837650Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4838204Z frame #19: <unknown function> + 0x1474b2 (0x5558ca0f04b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4838722Z frame #20: PyObject_Call + 0xbc (0x5558ca0f0e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4839239Z frame #21: <unknown function> + 0x1ff69d (0x5558ca1a869d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4839775Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5558ca0de023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4840341Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5558ca0d9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4840903Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4841560Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5558ca0d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4842178Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4842738Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5558ca0d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4843350Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4843911Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5558ca0d9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4844589Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5558ca0dd4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4845159Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5558ca0eeab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4845693Z frame #32: <unknown function> + 0x205949 (0x5558ca1ae949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4846238Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5558ca0de023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4846799Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5558ca0d9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4847361Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4847921Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5558ca0d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4848479Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4849037Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5558ca0d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4849596Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4850153Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5558ca0d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4850717Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4851282Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5558ca0d63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4851840Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4852546Z frame #44: <unknown function> + 0x7ab3f7 (0x7f21834523f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4853240Z frame #45: <unknown function> + 0x13bc88 (0x5558ca0e4c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4853764Z frame #46: PyObject_Call + 0x20f (0x5558ca0f0f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4854316Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5558ca0da98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4854871Z frame #48: <unknown function> + 0x147608 (0x5558ca0f0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4855428Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5558ca0d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4856101Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4856665Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5558ca0d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4857227Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4857787Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5558ca0d63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4858348Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4858902Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5558ca0d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4859457Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4860110Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5558ca0d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4860749Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4861313Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5558ca0d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4861945Z frame #60: <unknown function> + 0x147608 (0x5558ca0f0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4862506Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5558ca0d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4863138Z frame #62: <unknown function> + 0x147608 (0x5558ca0f0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4863448Z 
2025-07-17T11:07:26.4863546Z SIGABRT(6), PID: 1811295, Thread 1816149: 
2025-07-17T11:07:26.4864203Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4864978Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4865451Z frame #2: read + 0x4c (0x7ffab887481c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4865886Z frame #3: _Py_read + 0x4c (0x5556c2d11a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4866385Z frame #4: <unknown function> + 0x208c3b (0x5556c2d3fc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4866924Z frame #5: <unknown function> + 0x14648d (0x5556c2c7d48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4867493Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5556c2c8c215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4868049Z frame #7: <unknown function> + 0x208ae7 (0x5556c2d3fae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4868584Z frame #8: <unknown function> + 0x208472 (0x5556c2d3f472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4869119Z frame #9: <unknown function> + 0x1da6ee (0x5556c2d116ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4869662Z frame #10: <unknown function> + 0x1483bc (0x5556c2c7f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4870224Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4870787Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4871344Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5556c2c633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4871899Z frame #14: <unknown function> + 0x147734 (0x5556c2c7e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4872455Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5556c2c65c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4873016Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4873568Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4874128Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5556c2c730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4874678Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5556c2c637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4875225Z frame #20: <unknown function> + 0x147734 (0x5556c2c7e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4875755Z frame #21: <unknown function> + 0x21ca82 (0x5556c2d53a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4876289Z frame #22: <unknown function> + 0x21c9e4 (0x5556c2d539e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.4876811Z frame #23: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4877340Z frame #24: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4877737Z 
2025-07-17T11:07:26.4877831Z SIGABRT(6), PID: 1811296, Thread 1811343: 
2025-07-17T11:07:26.4878544Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4879322Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4879894Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4880408Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4881269Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4882070Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4882578Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4882977Z SIGABRT(6), PID: 1811299, Thread 1811668: 
2025-07-17T11:07:26.4883626Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4884388Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4884893Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4885391Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4886189Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4886987Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4887503Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4887900Z SIGABRT(6), PID: 1811297, Thread 1811530: 
2025-07-17T11:07:26.4888548Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4889304Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4889809Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4890307Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4891096Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4891885Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4892395Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4892699Z 
2025-07-17T11:07:26.4892793Z SIGABRT(6), PID: 1811300, Thread 1811768: 
2025-07-17T11:07:26.4893434Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4894195Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4894699Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4895194Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4896057Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4896953Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4897468Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4897856Z 
2025-07-17T11:07:26.4897860Z 
2025-07-17T11:07:26.4897863Z 
2025-07-17T11:07:26.4897956Z SIGABRT(6), PID: 1811299, Thread 1811670: 
2025-07-17T11:07:26.4898676Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4899556Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4900058Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4900555Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4901347Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4902142Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4902653Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4902963Z 
2025-07-17T11:07:26.4903056Z SIGABRT(6), PID: 1811297, Thread 1811532: 
2025-07-17T11:07:26.4903700Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4904473Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4904976Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4905472Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4906264Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4907086Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4907606Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4907914Z 
2025-07-17T11:07:26.4908015Z SIGABRT(6), PID: 1811296, Thread 1811345: 
2025-07-17T11:07:26.4908672Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4909442Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4909949Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4910452Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4911255Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4912051Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4912565Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4912872Z 
2025-07-17T11:07:26.4912964Z SIGABRT(6), PID: 1811300, Thread 1811769: 
2025-07-17T11:07:26.4913616Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4914377Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4914885Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4915479Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4916342Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4917143Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4917717Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4918094Z 
2025-07-17T11:07:26.4918187Z SIGABRT(6), PID: 1811299, Thread 1811672: 
2025-07-17T11:07:26.4918836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4919597Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4920102Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4920605Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4921417Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4922210Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4922722Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4923037Z 
2025-07-17T11:07:26.4923127Z SIGABRT(6), PID: 1811297, Thread 1811534: 
2025-07-17T11:07:26.4923769Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4924529Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4925031Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4925523Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4926323Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4927110Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4927621Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4927926Z 
2025-07-17T11:07:26.4928020Z SIGABRT(6), PID: 1811300, Thread 1811770: 
2025-07-17T11:07:26.4928664Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4929429Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4929937Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4930437Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4931227Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4932024Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4932539Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4932847Z 
2025-07-17T11:07:26.4932943Z SIGABRT(6), PID: 1811296, Thread 1811348: 
2025-07-17T11:07:26.4933584Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4934434Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4934999Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4935509Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4936458Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4937330Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4937842Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4938150Z 
2025-07-17T11:07:26.4938240Z SIGABRT(6), PID: 1811299, Thread 1811674: 
2025-07-17T11:07:26.4938889Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4939656Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4940160Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4940658Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4941446Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4942247Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4942758Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4943070Z 
2025-07-17T11:07:26.4943160Z SIGABRT(6), PID: 1811297, Thread 1811536: 
2025-07-17T11:07:26.4943804Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4944569Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4945073Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4945572Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4946357Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4947147Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4947655Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4947961Z 
2025-07-17T11:07:26.4948056Z SIGABRT(6), PID: 1811296, Thread 1811350: 
2025-07-17T11:07:26.4948706Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4949465Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4949970Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4950471Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4951257Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4952044Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4952651Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4952957Z 
2025-07-17T11:07:26.4953049Z SIGABRT(6), PID: 1811300, Thread 1811771: 
2025-07-17T11:07:26.4953769Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4954597Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4955103Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4955666Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4956452Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4957250Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4957763Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4958071Z 
2025-07-17T11:07:26.4958163Z SIGABRT(6), PID: 1811299, Thread 1811677: 
2025-07-17T11:07:26.4958808Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4959578Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4960085Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4960586Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4961376Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4962172Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4962683Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4962991Z 
2025-07-17T11:07:26.4963081Z SIGABRT(6), PID: 1811297, Thread 1811538: 
2025-07-17T11:07:26.4963720Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4964475Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4964978Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4965473Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4966261Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4967053Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4967562Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4967870Z 
2025-07-17T11:07:26.4967961Z SIGABRT(6), PID: 1811299, Thread 1811680: 
2025-07-17T11:07:26.4968610Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4969381Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4969893Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4970390Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4971253Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4972106Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4972626Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4972932Z 
2025-07-17T11:07:26.4973111Z SIGABRT(6), PID: 1811300, Thread 1811772: 
2025-07-17T11:07:26.4973755Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4974587Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4975088Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4975585Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4976447Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4977238Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4977747Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4978060Z 
2025-07-17T11:07:26.4978152Z SIGABRT(6), PID: 1811295, Thread 1816200: 
2025-07-17T11:07:26.4978812Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4979581Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4980088Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4980584Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4981263Z frame #4: <unknown function> + 0x5a02023 (0x7ff9fcc1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4982309Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9fcc201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4983561Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9fcc171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.4984635Z frame #7: <unknown function> + 0x79d655 (0x7ffa0e107655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.4985349Z frame #8: <unknown function> + 0xdc253 (0x7ff9c5152253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.4985898Z frame #9: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4986429Z frame #10: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4986745Z 
2025-07-17T11:07:26.4986838Z SIGABRT(6), PID: 1811296, Thread 1811351: 
2025-07-17T11:07:26.4987492Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4988267Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4988777Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4989277Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4990081Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4991051Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4991580Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4991888Z 
2025-07-17T11:07:26.4991984Z SIGABRT(6), PID: 1811297, Thread 1811540: 
2025-07-17T11:07:26.4992712Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4993554Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4994060Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4994562Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4995362Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.4996161Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4996676Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4996983Z 
2025-07-17T11:07:26.4997078Z SIGABRT(6), PID: 1811299, Thread 1811682: 
2025-07-17T11:07:26.4997722Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.4998486Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4998990Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.4999484Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5000276Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5001072Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5001588Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5001902Z 
2025-07-17T11:07:26.5001992Z SIGABRT(6), PID: 1811300, Thread 1811773: 
2025-07-17T11:07:26.5002640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5003399Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5003904Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5004402Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5005196Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5005995Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5006511Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5006821Z 
2025-07-17T11:07:26.5006911Z SIGABRT(6), PID: 1811297, Thread 1811543: 
2025-07-17T11:07:26.5007550Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5008311Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5008901Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5009398Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5010254Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5011110Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5011631Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5012002Z 
2025-07-17T11:07:26.5012096Z SIGABRT(6), PID: 1811296, Thread 1811354: 
2025-07-17T11:07:26.5012737Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5013498Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5014003Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5014503Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5015297Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5016161Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5016673Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5016977Z 
2025-07-17T11:07:26.5017072Z SIGABRT(6), PID: 1811299, Thread 1811686: 
2025-07-17T11:07:26.5017717Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5018487Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5018996Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5019488Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5020282Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5021078Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5021594Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5021909Z 
2025-07-17T11:07:26.5021998Z SIGABRT(6), PID: 1811300, Thread 1811774: 
2025-07-17T11:07:26.5022642Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5023401Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5023910Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5024411Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5025207Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5026005Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5026515Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5026822Z 
2025-07-17T11:07:26.5026912Z SIGABRT(6), PID: 1811297, Thread 1811546: 
2025-07-17T11:07:26.5027657Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5028489Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5028998Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5029591Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5030390Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5031267Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5031778Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5032087Z 
2025-07-17T11:07:26.5032181Z SIGABRT(6), PID: 1811296, Thread 1811356: 
2025-07-17T11:07:26.5032820Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5033583Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5034088Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5034589Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5035378Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5036175Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5036685Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5036994Z 
2025-07-17T11:07:26.5037087Z SIGABRT(6), PID: 1811299, Thread 1811689: 
2025-07-17T11:07:26.5037730Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5038492Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5038999Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5039493Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5040280Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5041077Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5041594Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5041904Z 
2025-07-17T11:07:26.5041994Z SIGABRT(6), PID: 1811300, Thread 1811775: 
2025-07-17T11:07:26.5042640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5043399Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5043905Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5044403Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5045194Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5046073Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5046588Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5046896Z 
2025-07-17T11:07:26.5047075Z SIGABRT(6), PID: 1811299, Thread 1811692: 
2025-07-17T11:07:26.5047721Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5048548Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5049115Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5049614Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5050402Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5051198Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5051713Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5052021Z 
2025-07-17T11:07:26.5052114Z SIGABRT(6), PID: 1811297, Thread 1811549: 
2025-07-17T11:07:26.5052760Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5053523Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5054026Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5054524Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5055312Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5056191Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5056708Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5057103Z SIGABRT(6), PID: 1811296, Thread 1811359: 
2025-07-17T11:07:26.5057746Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5058506Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5059013Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5059510Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5060292Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5061091Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5061602Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5061999Z SIGABRT(6), PID: 1811299, Thread 1811694: 
2025-07-17T11:07:26.5062645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5062859Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5063064Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5063264Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5063858Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5064144Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5064361Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5064527Z SIGABRT(6), PID: 1811300, Thread 1811776: 
2025-07-17T11:07:26.5065013Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5065306Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5065513Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5065718Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5066225Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5066441Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5066651Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5066657Z 
2025-07-17T11:07:26.5066661Z 
2025-07-17T11:07:26.5066667Z 
2025-07-17T11:07:26.5066670Z 
2025-07-17T11:07:26.5066762Z SIGABRT(6), PID: 1811299, Thread 1811696: 
2025-07-17T11:07:26.5067238Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5067445Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5067656Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5067858Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5068363Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5068571Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5068783Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5068788Z 
2025-07-17T11:07:26.5068879Z SIGABRT(6), PID: 1811297, Thread 1811552: 
2025-07-17T11:07:26.5069351Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5069561Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5069770Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5069971Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5070472Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5070680Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5070892Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5070896Z 
2025-07-17T11:07:26.5070986Z SIGABRT(6), PID: 1811300, Thread 1811777: 
2025-07-17T11:07:26.5071454Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5071735Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5071943Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5072202Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5072759Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5072970Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5073242Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5073246Z 
2025-07-17T11:07:26.5073335Z SIGABRT(6), PID: 1811296, Thread 1811360: 
2025-07-17T11:07:26.5073808Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5074015Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5074223Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5074422Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5074922Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5075131Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5075339Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5075343Z 
2025-07-17T11:07:26.5075433Z SIGABRT(6), PID: 1811295, Thread 1816201: 
2025-07-17T11:07:26.5075906Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5076115Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5076323Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5076523Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5076905Z frame #4: <unknown function> + 0x5a02023 (0x7ff9fcc1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5077480Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9fcc201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5078076Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9fcc171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5078469Z frame #7: <unknown function> + 0x79d655 (0x7ffa0e107655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5078703Z frame #8: <unknown function> + 0xdc253 (0x7ff9c5152253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5078925Z frame #9: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5079145Z frame #10: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5079151Z 
2025-07-17T11:07:26.5079247Z SIGABRT(6), PID: 1811299, Thread 1811698: 
2025-07-17T11:07:26.5079725Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5080025Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5080235Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5080501Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5081076Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5081292Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5081564Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5081568Z 
2025-07-17T11:07:26.5081663Z SIGABRT(6), PID: 1811297, Thread 1811555: 
2025-07-17T11:07:26.5082137Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5082347Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5082553Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5082754Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5083252Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5083463Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5083672Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5083676Z 
2025-07-17T11:07:26.5083770Z SIGABRT(6), PID: 1811300, Thread 1811778: 
2025-07-17T11:07:26.5084239Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5084449Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5084654Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5084855Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5085353Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5085563Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5085773Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5085867Z SIGABRT(6), PID: 1811296, Thread 1811362: 
2025-07-17T11:07:26.5086346Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5086556Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5086761Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5086968Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5087463Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5087674Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5087879Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5087886Z 
2025-07-17T11:07:26.5087982Z 
2025-07-17T11:07:26.5088075Z SIGABRT(6), PID: 1811299, Thread 1811700: 
2025-07-17T11:07:26.5088607Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5088819Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5089027Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5089409Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5089969Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5090177Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5090386Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5090392Z 
2025-07-17T11:07:26.5090480Z SIGABRT(6), PID: 1811297, Thread 1811558: 
2025-07-17T11:07:26.5090955Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5091161Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5091367Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5091566Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5092071Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5092276Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5092489Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5092493Z 
2025-07-17T11:07:26.5092582Z SIGABRT(6), PID: 1811300, Thread 1811779: 
2025-07-17T11:07:26.5093056Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5093263Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5093474Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5093672Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5094167Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5094377Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5094588Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5094592Z 
2025-07-17T11:07:26.5094682Z SIGABRT(6), PID: 1811296, Thread 1811364: 
2025-07-17T11:07:26.5095157Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5095364Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5095572Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5095770Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5096335Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5096636Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5096922Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5096928Z 
2025-07-17T11:07:26.5097019Z SIGABRT(6), PID: 1811299, Thread 1811703: 
2025-07-17T11:07:26.5097566Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5104544Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5104967Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5105198Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5105738Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5105958Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5106176Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5106182Z 
2025-07-17T11:07:26.5106284Z SIGABRT(6), PID: 1811297, Thread 1811561: 
2025-07-17T11:07:26.5106790Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5107009Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5107210Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5107413Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5107927Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5108136Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5108343Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5108348Z 
2025-07-17T11:07:26.5108446Z SIGABRT(6), PID: 1811300, Thread 1811780: 
2025-07-17T11:07:26.5108916Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5109125Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5109327Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5109528Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5110028Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5110233Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5110442Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5110449Z 
2025-07-17T11:07:26.5110542Z SIGABRT(6), PID: 1811296, Thread 1811366: 
2025-07-17T11:07:26.5111015Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5111219Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5111418Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5111700Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5112251Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5112465Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5112731Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5112826Z SIGABRT(6), PID: 1811299, Thread 1811705: 
2025-07-17T11:07:26.5113353Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5113560Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5113758Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5113956Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5114449Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5114654Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5114860Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5114867Z 
2025-07-17T11:07:26.5114871Z 
2025-07-17T11:07:26.5114960Z SIGABRT(6), PID: 1811297, Thread 1811564: 
2025-07-17T11:07:26.5115425Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5115629Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5115829Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5116027Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5116514Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5116719Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5116929Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5116933Z 
2025-07-17T11:07:26.5117019Z SIGABRT(6), PID: 1811300, Thread 1811781: 
2025-07-17T11:07:26.5117486Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5117693Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5117893Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5118089Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5118586Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5118788Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5118997Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5119001Z 
2025-07-17T11:07:26.5119088Z SIGABRT(6), PID: 1811299, Thread 1811707: 
2025-07-17T11:07:26.5119553Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5119835Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5120098Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5120293Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5120873Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5121138Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5121348Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5121437Z SIGABRT(6), PID: 1811297, Thread 1811567: 
2025-07-17T11:07:26.5121901Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5122106Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5122309Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5122499Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5122991Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5123194Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5123400Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5123404Z 
2025-07-17T11:07:26.5123407Z 
2025-07-17T11:07:26.5123494Z SIGABRT(6), PID: 1811296, Thread 1811368: 
2025-07-17T11:07:26.5123958Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5124162Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5124365Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5124559Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5125051Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5125253Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5125458Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5125464Z 
2025-07-17T11:07:26.5125551Z SIGABRT(6), PID: 1811300, Thread 1811782: 
2025-07-17T11:07:26.5126016Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5126220Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5126422Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5126616Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5127111Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5127318Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5127602Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5127607Z 
2025-07-17T11:07:26.5127696Z SIGABRT(6), PID: 1811299, Thread 1811709: 
2025-07-17T11:07:26.5128227Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5128496Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5128701Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5128958Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5129456Z frame #4: <unknown function> + 0x3475ab (0x7f6e1c7b15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5129661Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5129868Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5129873Z 
2025-07-17T11:07:26.5129964Z SIGABRT(6), PID: 1811297, Thread 1811570: 
2025-07-17T11:07:26.5130426Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5130631Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5130832Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5131027Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5131517Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5131724Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5131933Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5131937Z 
2025-07-17T11:07:26.5132025Z SIGABRT(6), PID: 1811296, Thread 1811370: 
2025-07-17T11:07:26.5132488Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5132692Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5132893Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5133088Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5133569Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5133779Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5133984Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5133989Z 
2025-07-17T11:07:26.5134079Z SIGABRT(6), PID: 1811300, Thread 1811783: 
2025-07-17T11:07:26.5134541Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5134748Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5134946Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5135141Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5135716Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5136110Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5136343Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5136436Z SIGABRT(6), PID: 1811299, Thread 1812364: 
2025-07-17T11:07:26.5136982Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5137515Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f6d2e01108a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5137724Z frame #2: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5137917Z frame #3: pthread_kill + 0x12c (0x7f6e1f91b9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5138081Z frame #4: raise + 0x16 (0x7f6e1f8c7476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5138248Z frame #5: abort + 0xd3 (0x7f6e1f8ad7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5138486Z frame #6: <unknown function> + 0x972cc (0x7f6c9bb9a2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5138717Z frame #7: <unknown function> + 0x95f47 (0x7f6c9bb98f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5138939Z frame #8: <unknown function> + 0x34b71 (0x7f6c9bb37b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5139148Z frame #9: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5139359Z frame #10: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5139367Z 
2025-07-17T11:07:26.5139371Z 
2025-07-17T11:07:26.5139460Z SIGABRT(6), PID: 1811295, Thread 1816202: 
2025-07-17T11:07:26.5139934Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5140144Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5140347Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5140548Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5140940Z frame #4: <unknown function> + 0x5a02023 (0x7ff9fcc1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5141526Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9fcc201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5142124Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9fcc171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5142519Z frame #7: <unknown function> + 0x79d655 (0x7ffa0e107655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5142757Z frame #8: <unknown function> + 0xdc253 (0x7ff9c5152253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5142971Z frame #9: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5143188Z frame #10: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5143192Z 
2025-07-17T11:07:26.5143283Z SIGABRT(6), PID: 1811297, Thread 1811573: 
2025-07-17T11:07:26.5143766Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5144059Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5144264Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5144524Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5145095Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5145305Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5145579Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5145583Z 
2025-07-17T11:07:26.5145671Z SIGABRT(6), PID: 1811296, Thread 1811372: 
2025-07-17T11:07:26.5146145Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5146354Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5146564Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5146760Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5147258Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5147463Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5147676Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5147680Z 
2025-07-17T11:07:26.5147768Z SIGABRT(6), PID: 1811300, Thread 1811784: 
2025-07-17T11:07:26.5148231Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5148439Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5148641Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5148836Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5149329Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5149538Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5149746Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5149751Z 
2025-07-17T11:07:26.5149838Z SIGABRT(6), PID: 1811299, Thread 1816088: 
2025-07-17T11:07:26.5150301Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5150509Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5150674Z frame #2: __poll + 0x4f (0x7f6e1f99dbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5151087Z frame #3: <unknown function> + 0x489f (0x7f6e1f4a989f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.5151314Z frame #4: <unknown function> + 0x1483bc (0x55560df393bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5151565Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5151790Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5152104Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5152324Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5152651Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55560df21a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5152881Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5153189Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55560df1fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5153471Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5153711Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5153932Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5154165Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5154383Z frame #16: <unknown function> + 0x147734 (0x55560df38734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5154601Z frame #17: <unknown function> + 0x21ca82 (0x55560e00da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5154819Z frame #18: <unknown function> + 0x21c9e4 (0x55560e00d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5155039Z frame #19: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5155253Z frame #20: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5155258Z 
2025-07-17T11:07:26.5155350Z SIGABRT(6), PID: 1811297, Thread 1811576: 
2025-07-17T11:07:26.5155836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5156050Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5156251Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5156452Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5156959Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5157169Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5157380Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5157384Z 
2025-07-17T11:07:26.5157474Z SIGABRT(6), PID: 1811300, Thread 1811785: 
2025-07-17T11:07:26.5157939Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5158147Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5158348Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5158547Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5159039Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5159248Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5159457Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5159461Z 
2025-07-17T11:07:26.5159550Z SIGABRT(6), PID: 1811296, Thread 1811374: 
2025-07-17T11:07:26.5160013Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5160357Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5160564Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5160762Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5161310Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5161579Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5161788Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5161792Z 
2025-07-17T11:07:26.5161881Z SIGABRT(6), PID: 1811300, Thread 1811786: 
2025-07-17T11:07:26.5162347Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5162555Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5162754Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5162954Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5163440Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5163647Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5163854Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5163863Z 
2025-07-17T11:07:26.5163948Z SIGABRT(6), PID: 1811296, Thread 1811376: 
2025-07-17T11:07:26.5164417Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5164621Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5164827Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5165023Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5165513Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5165717Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5165925Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5165931Z 
2025-07-17T11:07:26.5166017Z SIGABRT(6), PID: 1811297, Thread 1811579: 
2025-07-17T11:07:26.5166484Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5166686Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5166892Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5167087Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5167577Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5167781Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5168070Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5168074Z 
2025-07-17T11:07:26.5168160Z SIGABRT(6), PID: 1811299, Thread 1816089: 
2025-07-17T11:07:26.5168687Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5168954Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5169162Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5169442Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6e1f9192dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5169886Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f6d59d59048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5170304Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f6d59d5bbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5170538Z frame #6: <unknown function> + 0xdc253 (0x7f6d2c27d253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5170741Z frame #7: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5170952Z frame #8: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5170957Z 
2025-07-17T11:07:26.5171043Z SIGABRT(6), PID: 1811300, Thread 1811787: 
2025-07-17T11:07:26.5171508Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5171711Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5171911Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5172108Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5172605Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5172807Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5173021Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5173026Z 
2025-07-17T11:07:26.5173114Z SIGABRT(6), PID: 1811296, Thread 1811377: 
2025-07-17T11:07:26.5173579Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5173781Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5173989Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5174182Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5174677Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5174887Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5175092Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5175097Z 
2025-07-17T11:07:26.5175185Z SIGABRT(6), PID: 1811297, Thread 1811582: 
2025-07-17T11:07:26.5175643Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5175847Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5176196Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5176474Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5176969Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5177252Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5177527Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5177532Z 
2025-07-17T11:07:26.5177620Z SIGABRT(6), PID: 1811300, Thread 1811788: 
2025-07-17T11:07:26.5178082Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5178294Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5178495Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5178696Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5179194Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5179400Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5179612Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5179615Z 
2025-07-17T11:07:26.5179703Z SIGABRT(6), PID: 1811296, Thread 1811380: 
2025-07-17T11:07:26.5180162Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5180368Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5180569Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5180769Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5181259Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5181466Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5181668Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5181673Z 
2025-07-17T11:07:26.5181760Z SIGABRT(6), PID: 1811294, Thread 1811294: 
2025-07-17T11:07:26.5182220Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5182429Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5182658Z frame #2: <unknown function> + 0x7743a (0x7f7a5316f43a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5182887Z frame #3: <unknown function> + 0x76c8e (0x7f7a5316ec8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5183112Z frame #4: <unknown function> + 0x6ad69 (0x7f7a53162d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5183335Z frame #5: <unknown function> + 0x3f54db (0x7f7ae3dc74db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5183545Z frame #6: <unknown function> + 0x3e11d8 (0x7f7ae3db31d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5183755Z frame #7: <unknown function> + 0xb2f86 (0x7f7ae3a84f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5184070Z frame #8: <unknown function> + 0x8e501 (0x7f7ae3a60501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5184527Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f7b0e8677ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.5184921Z frame #10: <unknown function> + 0xd307d9 (0x7f7b2cdba7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5185211Z frame #11: <unknown function> + 0x139902 (0x56060cda3902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5185522Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x56060cd9aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5185752Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5185994Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x56060cd9aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5186221Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5186458Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5186687Z frame #17: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5186918Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5187141Z frame #19: <unknown function> + 0x1474b2 (0x56060cdb14b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5187343Z frame #20: PyObject_Call + 0xbc (0x56060cdb1e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5187564Z frame #21: <unknown function> + 0x1ff69d (0x56060ce6969d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5187785Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x56060cd9f023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5188027Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x56060cd9acc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5188249Z frame #24: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5188486Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x56060cd963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5188705Z frame #26: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5188940Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x56060cd963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5189160Z frame #28: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5189402Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x56060cd9aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5189655Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x56060cd9e4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5189874Z frame #31: _PyObject_Call_Prepend + 0x69 (0x56060cdafab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5190093Z frame #32: <unknown function> + 0x205949 (0x56060ce6f949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5190316Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x56060cd9f023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5190554Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x56060cd9acc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5190778Z frame #35: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5191014Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x56060cd963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5191241Z frame #37: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5191473Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x56060cd963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5191696Z frame #39: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5192003Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x56060cd963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5192283Z frame #41: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5192525Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x56060cd973f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5192806Z frame #43: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5193203Z frame #44: <unknown function> + 0x7ab3f7 (0x7f7b2c8353f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5193485Z frame #45: <unknown function> + 0x13bc88 (0x56060cda5c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5193691Z frame #46: PyObject_Call + 0x20f (0x56060cdb1f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5193937Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x56060cd9b98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5194154Z frame #48: <unknown function> + 0x147608 (0x56060cdb1608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5194396Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x56060cd98c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5194618Z frame #50: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5194856Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x56060cd963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5195075Z frame #52: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5195315Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x56060cd973f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5195536Z frame #54: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5195769Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x56060cd963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5195993Z frame #56: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5196240Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x56060cd98c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5196464Z frame #58: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5196700Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x56060cd98c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5196913Z frame #60: <unknown function> + 0x147608 (0x56060cdb1608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5197151Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x56060cd98c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5197366Z frame #62: <unknown function> + 0x147608 (0x56060cdb1608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5197459Z SIGABRT(6), PID: 1811297, Thread 1811585: 
2025-07-17T11:07:26.5197936Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5198152Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5198356Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5198557Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5199063Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5199274Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5199480Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5199555Z 
2025-07-17T11:07:26.5199563Z 
2025-07-17T11:07:26.5199653Z SIGABRT(6), PID: 1811300, Thread 1811793: 
2025-07-17T11:07:26.5200184Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5200397Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5200663Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5200865Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5201423Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5201627Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5201836Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5201843Z 
2025-07-17T11:07:26.5201930Z SIGABRT(6), PID: 1811299, Thread 1816090: 
2025-07-17T11:07:26.5202403Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5202607Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5202813Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5203046Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6e1f9192dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5203516Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f6d59d53741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5203743Z frame #5: <unknown function> + 0xdc253 (0x7f6d2c27d253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5203953Z frame #6: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5204161Z frame #7: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5204165Z 
2025-07-17T11:07:26.5204255Z SIGABRT(6), PID: 1811297, Thread 1811587: 
2025-07-17T11:07:26.5204722Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5204925Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5205126Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5205323Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5205819Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5206026Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5206233Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5206238Z 
2025-07-17T11:07:26.5206327Z SIGABRT(6), PID: 1811296, Thread 1811382: 
2025-07-17T11:07:26.5206790Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5206998Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5207197Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5207395Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5207964Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5208230Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5208441Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5208445Z 
2025-07-17T11:07:26.5208594Z SIGABRT(6), PID: 1811299, Thread 1816093: 
2025-07-17T11:07:26.5209067Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5209355Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5209521Z frame #2: ioctl + 0x3f (0x7f6e1f99f94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5209759Z frame #3: <unknown function> + 0x13eef0 (0x7f6c9bc41ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5209988Z frame #4: <unknown function> + 0x1378c4 (0x7f6c9bc3a8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5210224Z frame #5: <unknown function> + 0x95b2b (0x7f6c9bb98b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5210449Z frame #6: <unknown function> + 0x34b71 (0x7f6c9bb37b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5210661Z frame #7: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5210869Z frame #8: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5210875Z 
2025-07-17T11:07:26.5210964Z SIGABRT(6), PID: 1811295, Thread 1816203: 
2025-07-17T11:07:26.5211433Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5211640Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5211846Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5212046Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5212424Z frame #4: <unknown function> + 0x5a02023 (0x7ff9fcc1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5212993Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9fcc201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5213588Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9fcc171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5213975Z frame #7: <unknown function> + 0x79d655 (0x7ffa0e107655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5214206Z frame #8: <unknown function> + 0xdc253 (0x7ff9c5152253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5214415Z frame #9: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5214627Z frame #10: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5214716Z SIGABRT(6), PID: 1811294, Thread 1811304: 
2025-07-17T11:07:26.5215190Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5215398Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5215602Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5215798Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5216537Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5216750Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5216961Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5217037Z 
2025-07-17T11:07:26.5217127Z SIGABRT(6), PID: 1811300, Thread 1811794: 
2025-07-17T11:07:26.5217671Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5217875Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5218076Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5218273Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5218767Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5218971Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5219186Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5219190Z 
2025-07-17T11:07:26.5219195Z 
2025-07-17T11:07:26.5219283Z SIGABRT(6), PID: 1811297, Thread 1811590: 
2025-07-17T11:07:26.5219750Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5219953Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5220157Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5220351Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5220845Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5221050Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5221261Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5221267Z 
2025-07-17T11:07:26.5221354Z SIGABRT(6), PID: 1811300, Thread 1811795: 
2025-07-17T11:07:26.5221825Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5222031Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5222233Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5222431Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5222925Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5223129Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5223337Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5223341Z 
2025-07-17T11:07:26.5223430Z SIGABRT(6), PID: 1811296, Thread 1811385: 
2025-07-17T11:07:26.5223892Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5224306Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5224565Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5224767Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5225318Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5225584Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5225789Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5225793Z 
2025-07-17T11:07:26.5225882Z SIGABRT(6), PID: 1811299, Thread 1816107: 
2025-07-17T11:07:26.5226348Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5226556Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5226724Z frame #2: __poll + 0x4f (0x7f6e1f99dbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5226938Z frame #3: <unknown function> + 0xd4540 (0x7f6ce3db1540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5227141Z frame #4: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5227349Z frame #5: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5227355Z 
2025-07-17T11:07:26.5227440Z SIGABRT(6), PID: 1811294, Thread 1811305: 
2025-07-17T11:07:26.5227900Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5228107Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5228308Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5228503Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5229003Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5229208Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5229423Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5229427Z 
2025-07-17T11:07:26.5229513Z SIGABRT(6), PID: 1811297, Thread 1811593: 
2025-07-17T11:07:26.5229978Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5230182Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5230389Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5230582Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5231079Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5231283Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5231491Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5231496Z 
2025-07-17T11:07:26.5231586Z SIGABRT(6), PID: 1811299, Thread 1816115: 
2025-07-17T11:07:26.5232052Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5232331Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5232564Z frame #2: __poll + 0x4f (0x7f6e1f99dbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5232780Z frame #3: <unknown function> + 0xd65a2 (0x7f6ce3db35a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5233045Z frame #4: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5233314Z frame #5: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5233321Z 
2025-07-17T11:07:26.5233411Z SIGABRT(6), PID: 1811300, Thread 1811796: 
2025-07-17T11:07:26.5233881Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5234088Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5234290Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5234486Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5234979Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5235182Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5235393Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5235397Z 
2025-07-17T11:07:26.5235482Z SIGABRT(6), PID: 1811296, Thread 1811390: 
2025-07-17T11:07:26.5235948Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5236152Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5236357Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5236550Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5237046Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5237251Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5237456Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5237460Z 
2025-07-17T11:07:26.5237545Z SIGABRT(6), PID: 1811297, Thread 1811596: 
2025-07-17T11:07:26.5238014Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5238219Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5238421Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5238620Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5239129Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5239338Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5239549Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5239553Z 
2025-07-17T11:07:26.5239640Z SIGABRT(6), PID: 1811294, Thread 1811306: 
2025-07-17T11:07:26.5240186Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5240454Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5240666Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5240943Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5241448Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5241720Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5241932Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5241939Z 
2025-07-17T11:07:26.5242026Z SIGABRT(6), PID: 1811299, Thread 1816122: 
2025-07-17T11:07:26.5242492Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5242697Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5242902Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5243097Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5243318Z frame #4: <unknown function> + 0xd1e7f (0x7f6ce3daee7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5243523Z frame #5: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5243732Z frame #6: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5243738Z 
2025-07-17T11:07:26.5243824Z SIGABRT(6), PID: 1811300, Thread 1811797: 
2025-07-17T11:07:26.5244294Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5244500Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5244701Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5244897Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5245395Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5245602Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5245809Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5245815Z 
2025-07-17T11:07:26.5245902Z SIGABRT(6), PID: 1811296, Thread 1811392: 
2025-07-17T11:07:26.5246369Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5246574Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5246783Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5246981Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5247472Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5247677Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5247961Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5248051Z SIGABRT(6), PID: 1811297, Thread 1811599: 
2025-07-17T11:07:26.5248571Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5248844Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5249047Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5249310Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5249804Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5250009Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5250216Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5250306Z SIGABRT(6), PID: 1811294, Thread 1811307: 
2025-07-17T11:07:26.5250765Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5250972Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5251172Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5251370Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5251864Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5252072Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5252278Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5252371Z SIGABRT(6), PID: 1811300, Thread 1811798: 
2025-07-17T11:07:26.5252836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5253042Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5253245Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5253445Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5253935Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5254144Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5254352Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5254441Z SIGABRT(6), PID: 1811299, Thread 1816129: 
2025-07-17T11:07:26.5254902Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5255106Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5255309Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5255517Z frame #3: <unknown function> + 0x9ca53 (0x7f6e1f921a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5255768Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55560df05bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5256154Z frame #5: <unknown function> + 0x1d314f (0x55560dfc414f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5256461Z frame #6: <unknown function> + 0x1d2f6b (0x55560dfc3f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5256687Z frame #7: <unknown function> + 0x6df36 (0x55560de5ef36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5256926Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5257222Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5257535Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5257765Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5258001Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5258232Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5258466Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5258689Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5258919Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5259140Z frame #17: <unknown function> + 0x147734 (0x55560df38734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5259356Z frame #18: <unknown function> + 0x21ca82 (0x55560e00da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5259571Z frame #19: <unknown function> + 0x21c9e4 (0x55560e00d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5259782Z frame #20: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5259996Z frame #21: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5260000Z 
2025-07-17T11:07:26.5260004Z 
2025-07-17T11:07:26.5260008Z 
2025-07-17T11:07:26.5260011Z 
2025-07-17T11:07:26.5260014Z 
2025-07-17T11:07:26.5260108Z SIGABRT(6), PID: 1811297, Thread 1811601: 
2025-07-17T11:07:26.5260584Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5260796Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5261001Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5261199Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5261699Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5261910Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5262119Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5262123Z 
2025-07-17T11:07:26.5262214Z SIGABRT(6), PID: 1811295, Thread 1816204: 
2025-07-17T11:07:26.5262683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5262892Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5263093Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5263296Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5263672Z frame #4: <unknown function> + 0x5a02023 (0x7ff9fcc1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5264383Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9fcc201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5265036Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9fcc171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5265424Z frame #7: <unknown function> + 0x79d655 (0x7ffa0e107655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5265714Z frame #8: <unknown function> + 0xdc253 (0x7ff9c5152253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5265923Z frame #9: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5266136Z frame #10: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5266227Z SIGABRT(6), PID: 1811296, Thread 1811395: 
2025-07-17T11:07:26.5266705Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5266913Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5267115Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5267313Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5267812Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5268019Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5268227Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5268316Z SIGABRT(6), PID: 1811300, Thread 1811799: 
2025-07-17T11:07:26.5268793Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5269003Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5269209Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5269407Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5269896Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5270102Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5270312Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5270317Z 
2025-07-17T11:07:26.5270323Z 
2025-07-17T11:07:26.5270328Z 
2025-07-17T11:07:26.5270414Z SIGABRT(6), PID: 1811294, Thread 1811308: 
2025-07-17T11:07:26.5270882Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5271085Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5271289Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5271485Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5271981Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5272259Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5272554Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5272560Z 
2025-07-17T11:07:26.5272651Z SIGABRT(6), PID: 1811299, Thread 1816141: 
2025-07-17T11:07:26.5273181Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5273449Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5273614Z frame #2: read + 0x4c (0x7f6e1f99981c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5273791Z frame #3: _Py_read + 0x4c (0x55560dfcba6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5274020Z frame #4: <unknown function> + 0x208c3b (0x55560dff9c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5274239Z frame #5: <unknown function> + 0x14648d (0x55560df3748d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5274484Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55560df46215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5274703Z frame #7: <unknown function> + 0x208ae7 (0x55560dff9ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5274922Z frame #8: <unknown function> + 0x208472 (0x55560dff9472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5275143Z frame #9: <unknown function> + 0x1da6ee (0x55560dfcb6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5275372Z frame #10: <unknown function> + 0x1483bc (0x55560df393bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5275614Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5275849Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5276087Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55560df1d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5276320Z frame #14: <unknown function> + 0x147734 (0x55560df38734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5276573Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55560df1fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5276800Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5277034Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5277256Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55560df2d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5277485Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55560df1d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5277722Z frame #20: <unknown function> + 0x147734 (0x55560df38734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5277944Z frame #21: <unknown function> + 0x21ca82 (0x55560e00da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5278161Z frame #22: <unknown function> + 0x21c9e4 (0x55560e00d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5278380Z frame #23: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5278591Z frame #24: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5278597Z 
2025-07-17T11:07:26.5278684Z SIGABRT(6), PID: 1811297, Thread 1811604: 
2025-07-17T11:07:26.5279164Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5279371Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5279576Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5279867Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5280423Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5280632Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5280904Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5280909Z 
2025-07-17T11:07:26.5281055Z SIGABRT(6), PID: 1811300, Thread 1811800: 
2025-07-17T11:07:26.5281525Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5281732Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5281947Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5282142Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5282642Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5282848Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5283060Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5283158Z SIGABRT(6), PID: 1811296, Thread 1811398: 
2025-07-17T11:07:26.5283621Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5283822Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5284028Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5284225Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5284717Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5284923Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5285129Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5285136Z 
2025-07-17T11:07:26.5285139Z 
2025-07-17T11:07:26.5285224Z SIGABRT(6), PID: 1811294, Thread 1811309: 
2025-07-17T11:07:26.5285690Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5285897Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5286097Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5286303Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5286803Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5287012Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5287221Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5287225Z 
2025-07-17T11:07:26.5287313Z SIGABRT(6), PID: 1811297, Thread 1811607: 
2025-07-17T11:07:26.5287774Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5288058Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5288321Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5288521Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5289073Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5289360Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5289569Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5289575Z 
2025-07-17T11:07:26.5289664Z SIGABRT(6), PID: 1811296, Thread 1811401: 
2025-07-17T11:07:26.5290128Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5290335Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5290536Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5290734Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5291225Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5291435Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5291639Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5291643Z 
2025-07-17T11:07:26.5291735Z SIGABRT(6), PID: 1811300, Thread 1811801: 
2025-07-17T11:07:26.5292194Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5292401Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5292603Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5292802Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5293294Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5293504Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5293710Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5293717Z 
2025-07-17T11:07:26.5293807Z SIGABRT(6), PID: 1811294, Thread 1811310: 
2025-07-17T11:07:26.5294272Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5294479Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5294680Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5294879Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5295374Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5295581Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5295863Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5295871Z 
2025-07-17T11:07:26.5296033Z SIGABRT(6), PID: 1811297, Thread 1811610: 
2025-07-17T11:07:26.5296577Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5296784Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5297059Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5297326Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5297822Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5298024Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5298235Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5298239Z 
2025-07-17T11:07:26.5298330Z SIGABRT(6), PID: 1811296, Thread 1811404: 
2025-07-17T11:07:26.5298793Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5298996Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5299203Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5299397Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5299892Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5300101Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5300311Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5300401Z SIGABRT(6), PID: 1811300, Thread 1811802: 
2025-07-17T11:07:26.5300863Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5301065Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5301270Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5301465Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5301958Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5302161Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5302371Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5302375Z 
2025-07-17T11:07:26.5302378Z 
2025-07-17T11:07:26.5302463Z SIGABRT(6), PID: 1811297, Thread 1811613: 
2025-07-17T11:07:26.5302928Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5303132Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5303333Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5303526Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5304137Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5304398Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5304609Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5304613Z 
2025-07-17T11:07:26.5304702Z SIGABRT(6), PID: 1811294, Thread 1811311: 
2025-07-17T11:07:26.5305231Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5305497Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5305706Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5305902Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5306402Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5306610Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5306819Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5306823Z 
2025-07-17T11:07:26.5306913Z SIGABRT(6), PID: 1811296, Thread 1811406: 
2025-07-17T11:07:26.5307375Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5307580Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5307781Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5307980Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5308473Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5308679Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5308887Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5308891Z 
2025-07-17T11:07:26.5308984Z SIGABRT(6), PID: 1811297, Thread 1811616: 
2025-07-17T11:07:26.5309445Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5309651Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5309850Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5310047Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5310539Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5310747Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5310955Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5310960Z 
2025-07-17T11:07:26.5311052Z SIGABRT(6), PID: 1811300, Thread 1811803: 
2025-07-17T11:07:26.5311515Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5311720Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5311995Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5312252Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5312747Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5313013Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5313282Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5313285Z 
2025-07-17T11:07:26.5313375Z SIGABRT(6), PID: 1811294, Thread 1811313: 
2025-07-17T11:07:26.5313835Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5314043Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5314242Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5314441Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5314934Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5315141Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5315347Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5315351Z 
2025-07-17T11:07:26.5315440Z SIGABRT(6), PID: 1811300, Thread 1811804: 
2025-07-17T11:07:26.5315901Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5316115Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5316318Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5316522Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5317012Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5317219Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5317429Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5317516Z SIGABRT(6), PID: 1811296, Thread 1811408: 
2025-07-17T11:07:26.5317984Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5318190Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5318394Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5318587Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5319077Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5319285Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5319493Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5319497Z 
2025-07-17T11:07:26.5319500Z 
2025-07-17T11:07:26.5319586Z SIGABRT(6), PID: 1811297, Thread 1811619: 
2025-07-17T11:07:26.5320129Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5320389Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5320596Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5320849Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5321344Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5321612Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5321823Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5321829Z 
2025-07-17T11:07:26.5321915Z SIGABRT(6), PID: 1811295, Thread 1816205: 
2025-07-17T11:07:26.5322383Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5322590Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5322795Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5322989Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5323365Z frame #4: <unknown function> + 0x5a02023 (0x7ff9fcc1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5323933Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9fcc201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5324527Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9fcc171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5324908Z frame #7: <unknown function> + 0x79d655 (0x7ffa0e107655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5325144Z frame #8: <unknown function> + 0xdc253 (0x7ff9c5152253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5325350Z frame #9: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5325566Z frame #10: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5325570Z 
2025-07-17T11:07:26.5325657Z SIGABRT(6), PID: 1811294, Thread 1811314: 
2025-07-17T11:07:26.5326124Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5326334Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5326536Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5326742Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5327247Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5327457Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5327666Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5327670Z 
2025-07-17T11:07:26.5327759Z SIGABRT(6), PID: 1811296, Thread 1811410: 
2025-07-17T11:07:26.5328309Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5328582Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5328792Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5329052Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5329557Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5329847Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5330055Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5330148Z SIGABRT(6), PID: 1811297, Thread 1811622: 
2025-07-17T11:07:26.5330617Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5330827Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5331027Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5331225Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5331717Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5331926Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5332131Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5332223Z SIGABRT(6), PID: 1811300, Thread 1811805: 
2025-07-17T11:07:26.5332685Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5332891Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5333087Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5333291Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5333786Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5333995Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5334201Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5334207Z 
2025-07-17T11:07:26.5334295Z SIGABRT(6), PID: 1811299, Thread 1816168: 
2025-07-17T11:07:26.5334761Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5334969Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5335169Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5335365Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5335738Z frame #4: <unknown function> + 0x5a02023 (0x7f6d63d47023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5336369Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6d63d4b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5337125Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6d63d421d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5337509Z frame #7: <unknown function> + 0x79d655 (0x7f6d75232655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5337812Z frame #8: <unknown function> + 0xdc253 (0x7f6d2c27d253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5338095Z frame #9: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5338311Z frame #10: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5338318Z 
2025-07-17T11:07:26.5338322Z 
2025-07-17T11:07:26.5338325Z 
2025-07-17T11:07:26.5338413Z SIGABRT(6), PID: 1811294, Thread 1811315: 
2025-07-17T11:07:26.5338889Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5339099Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5339303Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5339500Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5339999Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5340205Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5340417Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5340424Z 
2025-07-17T11:07:26.5340510Z SIGABRT(6), PID: 1811296, Thread 1811412: 
2025-07-17T11:07:26.5340981Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5341187Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5341388Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5341583Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5342079Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5342286Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5342494Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5342583Z SIGABRT(6), PID: 1811300, Thread 1811806: 
2025-07-17T11:07:26.5343050Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5343253Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5343456Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5343650Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5344145Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5344346Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5344634Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5344639Z 
2025-07-17T11:07:26.5344644Z 
2025-07-17T11:07:26.5344730Z SIGABRT(6), PID: 1811297, Thread 1811626: 
2025-07-17T11:07:26.5345260Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5345528Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5345735Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5346104Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5346603Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5346809Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5347028Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5347032Z 
2025-07-17T11:07:26.5347121Z SIGABRT(6), PID: 1811294, Thread 1811316: 
2025-07-17T11:07:26.5347586Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5347795Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5348000Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5348196Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5348690Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5348902Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5349108Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5349112Z 
2025-07-17T11:07:26.5349201Z SIGABRT(6), PID: 1811296, Thread 1811414: 
2025-07-17T11:07:26.5349665Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5349870Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5350071Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5350267Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5350755Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5350964Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5351170Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5351174Z 
2025-07-17T11:07:26.5351263Z SIGABRT(6), PID: 1811297, Thread 1811630: 
2025-07-17T11:07:26.5351727Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5351930Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5352130Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5352331Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5352900Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5353170Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5353380Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5353385Z 
2025-07-17T11:07:26.5353532Z SIGABRT(6), PID: 1811300, Thread 1811807: 
2025-07-17T11:07:26.5353997Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5354269Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5354469Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5354666Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5355159Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5355364Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5355570Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5355576Z 
2025-07-17T11:07:26.5355667Z SIGABRT(6), PID: 1811294, Thread 1811317: 
2025-07-17T11:07:26.5356128Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5356333Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5356541Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5356738Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5357232Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5357439Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5357645Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5357655Z 
2025-07-17T11:07:26.5357742Z SIGABRT(6), PID: 1811296, Thread 1811416: 
2025-07-17T11:07:26.5358206Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5358407Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5358612Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5358806Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5359299Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5359504Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5359711Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5359716Z 
2025-07-17T11:07:26.5359803Z SIGABRT(6), PID: 1811297, Thread 1811637: 
2025-07-17T11:07:26.5360271Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5360473Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5360750Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5361005Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5361504Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5361787Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5362055Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5362062Z 
2025-07-17T11:07:26.5362148Z SIGABRT(6), PID: 1811300, Thread 1811808: 
2025-07-17T11:07:26.5362615Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5362821Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5363023Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5363218Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5363715Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5363920Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5364131Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5364136Z 
2025-07-17T11:07:26.5364225Z SIGABRT(6), PID: 1811297, Thread 1811640: 
2025-07-17T11:07:26.5364686Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5364888Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5365093Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5365286Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5365777Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5365982Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5366189Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5366276Z SIGABRT(6), PID: 1811294, Thread 1811318: 
2025-07-17T11:07:26.5366742Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5366956Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5367160Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5367355Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5367854Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5368058Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5368268Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5368354Z SIGABRT(6), PID: 1811296, Thread 1811418: 
2025-07-17T11:07:26.5368893Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5369156Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5369363Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5369630Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5370128Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5370396Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5370607Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5370613Z 
2025-07-17T11:07:26.5370617Z 
2025-07-17T11:07:26.5370621Z 
2025-07-17T11:07:26.5370711Z SIGABRT(6), PID: 1811300, Thread 1811809: 
2025-07-17T11:07:26.5371176Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5371382Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5371583Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5371780Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5372273Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5372479Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5372686Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5372691Z 
2025-07-17T11:07:26.5372786Z SIGABRT(6), PID: 1811295, Thread 1816206: 
2025-07-17T11:07:26.5373252Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5373462Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5373664Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5373864Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5374236Z frame #4: <unknown function> + 0x5a02023 (0x7ff9fcc1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5374800Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9fcc201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5375396Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9fcc171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5375782Z frame #7: <unknown function> + 0x79d655 (0x7ffa0e107655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5376110Z frame #8: <unknown function> + 0xdc253 (0x7ff9c5152253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5376322Z frame #9: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5376545Z frame #10: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5376636Z SIGABRT(6), PID: 1811297, Thread 1811642: 
2025-07-17T11:07:26.5377193Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5377472Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5377676Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5377946Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5378446Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5378729Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5378936Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5378943Z 
2025-07-17T11:07:26.5378948Z 
2025-07-17T11:07:26.5379038Z SIGABRT(6), PID: 1811300, Thread 1811812: 
2025-07-17T11:07:26.5379506Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5379708Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5379910Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5380105Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5380600Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5380801Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5381009Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5381015Z 
2025-07-17T11:07:26.5381101Z SIGABRT(6), PID: 1811296, Thread 1811420: 
2025-07-17T11:07:26.5381572Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5381775Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5381980Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5382173Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5382670Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5382872Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5383082Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5383168Z SIGABRT(6), PID: 1811294, Thread 1811319: 
2025-07-17T11:07:26.5383634Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5383835Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5384043Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5384239Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5384736Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5384938Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5385225Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5385229Z 
2025-07-17T11:07:26.5385232Z 
2025-07-17T11:07:26.5385377Z SIGABRT(6), PID: 1811299, Thread 1816169: 
2025-07-17T11:07:26.5385849Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5386115Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5386379Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5386575Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5386951Z frame #4: <unknown function> + 0x5a02023 (0x7f6d63d47023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5387515Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6d63d4b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5388109Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6d63d421d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5388487Z frame #7: <unknown function> + 0x79d655 (0x7f6d75232655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5388718Z frame #8: <unknown function> + 0xdc253 (0x7f6d2c27d253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5388927Z frame #9: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5389139Z frame #10: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5389145Z 
2025-07-17T11:07:26.5389233Z SIGABRT(6), PID: 1811297, Thread 1811645: 
2025-07-17T11:07:26.5389698Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5389905Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5390111Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5390307Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5390808Z frame #4: <unknown function> + 0x3475ab (0x7f710983f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5391013Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5391222Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5391228Z 
2025-07-17T11:07:26.5391316Z SIGABRT(6), PID: 1811300, Thread 1811813: 
2025-07-17T11:07:26.5391778Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5391983Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5392185Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5392383Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5392874Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5393081Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5393366Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5393370Z 
2025-07-17T11:07:26.5393459Z SIGABRT(6), PID: 1811296, Thread 1811422: 
2025-07-17T11:07:26.5394000Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5394267Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5394470Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5394731Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5395226Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5395433Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5395637Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5395643Z 
2025-07-17T11:07:26.5395732Z SIGABRT(6), PID: 1811294, Thread 1811320: 
2025-07-17T11:07:26.5396198Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5396403Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5396606Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5396805Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5397300Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5397510Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5397720Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5397724Z 
2025-07-17T11:07:26.5397813Z SIGABRT(6), PID: 1811297, Thread 1812354: 
2025-07-17T11:07:26.5398278Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5398726Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f701b09d08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5398929Z frame #2: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5399119Z frame #3: pthread_kill + 0x12c (0x7f710c9a99fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5399286Z frame #4: raise + 0x16 (0x7f710c955476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5399445Z frame #5: abort + 0xd3 (0x7f710c93b7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5399682Z frame #6: <unknown function> + 0x972cc (0x7f6f88c262cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5399909Z frame #7: <unknown function> + 0x95f47 (0x7f6f88c24f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5400135Z frame #8: <unknown function> + 0x34b71 (0x7f6f88bc3b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5400347Z frame #9: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5400558Z frame #10: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5400565Z 
2025-07-17T11:07:26.5400651Z SIGABRT(6), PID: 1811300, Thread 1811814: 
2025-07-17T11:07:26.5401120Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5401401Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5401665Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5401864Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5402422Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5402686Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5402897Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5402901Z 
2025-07-17T11:07:26.5402988Z SIGABRT(6), PID: 1811296, Thread 1811424: 
2025-07-17T11:07:26.5403458Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5403660Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5403863Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5404058Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5404552Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5404757Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5404964Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5404968Z 
2025-07-17T11:07:26.5405055Z SIGABRT(6), PID: 1811294, Thread 1811321: 
2025-07-17T11:07:26.5405524Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5405729Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5405936Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5406131Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5406629Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5406833Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5407042Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5407048Z 
2025-07-17T11:07:26.5407134Z SIGABRT(6), PID: 1811300, Thread 1811815: 
2025-07-17T11:07:26.5407605Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5407808Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5408011Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5408204Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5408698Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5408902Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5409188Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5409274Z SIGABRT(6), PID: 1811297, Thread 1815078: 
2025-07-17T11:07:26.5409805Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5410010Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5410235Z frame #2: __poll + 0x4f (0x7f710ca2bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5410694Z frame #3: <unknown function> + 0x489f (0x7f710c53789f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.5410921Z frame #4: <unknown function> + 0x1483bc (0x55b2cbf243bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5411165Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5411392Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5411626Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5411853Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5412094Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55b2cbf0ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5412325Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5412568Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b2cbf0ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5412793Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5413025Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5413251Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5413482Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5413706Z frame #16: <unknown function> + 0x147734 (0x55b2cbf23734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5413923Z frame #17: <unknown function> + 0x21ca82 (0x55b2cbff8a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5414143Z frame #18: <unknown function> + 0x21c9e4 (0x55b2cbff89e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5414352Z frame #19: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5414565Z frame #20: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5414570Z 
2025-07-17T11:07:26.5414573Z 
2025-07-17T11:07:26.5414661Z SIGABRT(6), PID: 1811296, Thread 1811426: 
2025-07-17T11:07:26.5415135Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5415347Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5415553Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5415753Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5416320Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5416531Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5416744Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5416749Z 
2025-07-17T11:07:26.5416929Z SIGABRT(6), PID: 1811294, Thread 1811322: 
2025-07-17T11:07:26.5417394Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5417674Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5417877Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5418146Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5418714Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5418922Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5419129Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5419135Z 
2025-07-17T11:07:26.5419225Z SIGABRT(6), PID: 1811300, Thread 1811816: 
2025-07-17T11:07:26.5419690Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5419897Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5420099Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5420296Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5420789Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5420996Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5421206Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5421210Z 
2025-07-17T11:07:26.5421299Z SIGABRT(6), PID: 1811296, Thread 1811427: 
2025-07-17T11:07:26.5421761Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5421968Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5422168Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5422368Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5422856Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5423066Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5423276Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5423280Z 
2025-07-17T11:07:26.5423373Z SIGABRT(6), PID: 1811294, Thread 1811323: 
2025-07-17T11:07:26.5423836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5424049Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5424253Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5424450Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5424945Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5425251Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5425459Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5425532Z 
2025-07-17T11:07:26.5425620Z SIGABRT(6), PID: 1811300, Thread 1811817: 
2025-07-17T11:07:26.5426142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5426349Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5426611Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5426808Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5427311Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5427518Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5427728Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5427732Z 
2025-07-17T11:07:26.5427818Z SIGABRT(6), PID: 1811296, Thread 1811429: 
2025-07-17T11:07:26.5428296Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5428499Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5428707Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5428910Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5429427Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5429646Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5429870Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5429875Z 
2025-07-17T11:07:26.5429972Z SIGABRT(6), PID: 1811297, Thread 1815079: 
2025-07-17T11:07:26.5430462Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5430666Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5430873Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5431099Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f710c9a72dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5431555Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f7046de5048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5431987Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f7046de7bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5432229Z frame #6: <unknown function> + 0xdc253 (0x7f7019309253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5432432Z frame #7: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5432641Z frame #8: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5432728Z SIGABRT(6), PID: 1811294, Thread 1811324: 
2025-07-17T11:07:26.5433196Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5433474Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5433748Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5433954Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5434530Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5434745Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5435024Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5435028Z 
2025-07-17T11:07:26.5435032Z 
2025-07-17T11:07:26.5435123Z SIGABRT(6), PID: 1811300, Thread 1811818: 
2025-07-17T11:07:26.5435595Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5435808Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5436031Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5436238Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5436755Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5436964Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5437169Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5437173Z 
2025-07-17T11:07:26.5437262Z SIGABRT(6), PID: 1811296, Thread 1811431: 
2025-07-17T11:07:26.5437731Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5437958Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5438160Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5438359Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5438860Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5439081Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5439294Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5439300Z 
2025-07-17T11:07:26.5439417Z SIGABRT(6), PID: 1811295, Thread 1816207: 
2025-07-17T11:07:26.5439893Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff9c6ee5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5440100Z frame #1: <unknown function> + 0x42520 (0x7ffab87a2520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5440303Z frame #2: <unknown function> + 0x91117 (0x7ffab87f1117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5440500Z frame #3: pthread_cond_wait + 0x211 (0x7ffab87f3a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5440875Z frame #4: <unknown function> + 0x5a02023 (0x7ff9fcc1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5441458Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9fcc201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5442187Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9fcc171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5442584Z frame #7: <unknown function> + 0x79d655 (0x7ffa0e107655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5442874Z frame #8: <unknown function> + 0xdc253 (0x7ff9c5152253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5443096Z frame #9: <unknown function> + 0x94ac3 (0x7ffab87f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5443375Z frame #10: <unknown function> + 0x126850 (0x7ffab8886850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5443486Z SIGABRT(6), PID: 1811294, Thread 1811325: 
2025-07-17T11:07:26.5443970Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5444199Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5444400Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5444598Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5445095Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5445305Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5445522Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5445629Z SIGABRT(6), PID: 1811300, Thread 1811819: 
2025-07-17T11:07:26.5446095Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5446319Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5446527Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5446735Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5447244Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5447470Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5447687Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5447780Z SIGABRT(6), PID: 1811296, Thread 1811433: 
2025-07-17T11:07:26.5448244Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5448453Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5448650Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5448845Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5449336Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5449550Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5449755Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5449841Z SIGABRT(6), PID: 1811299, Thread 1816170: 
2025-07-17T11:07:26.5450382Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5450664Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5450871Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5451125Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5451504Z frame #4: <unknown function> + 0x5a02023 (0x7f6d63d47023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5452150Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6d63d4b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5452739Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6d63d421d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5453123Z frame #7: <unknown function> + 0x79d655 (0x7f6d75232655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5453354Z frame #8: <unknown function> + 0xdc253 (0x7f6d2c27d253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5453558Z frame #9: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5453774Z frame #10: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5453860Z SIGABRT(6), PID: 1811297, Thread 1815080: 
2025-07-17T11:07:26.5454325Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5454528Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5454731Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5454952Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f710c9a72dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5455423Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f7046ddf741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5455650Z frame #5: <unknown function> + 0xdc253 (0x7f7019309253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5455856Z frame #6: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5456127Z frame #7: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5456131Z 
2025-07-17T11:07:26.5456135Z 
2025-07-17T11:07:26.5456138Z 
2025-07-17T11:07:26.5456141Z 
2025-07-17T11:07:26.5456146Z 
2025-07-17T11:07:26.5456152Z 
2025-07-17T11:07:26.5456239Z SIGABRT(6), PID: 1811297, Thread 1815090: 
2025-07-17T11:07:26.5456710Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5456913Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5457080Z frame #2: ioctl + 0x3f (0x7f710ca2d94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5457313Z frame #3: <unknown function> + 0x13eef0 (0x7f6f88ccdef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5457547Z frame #4: <unknown function> + 0x1378c4 (0x7f6f88cc68c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5457775Z frame #5: <unknown function> + 0x95b2b (0x7f6f88c24b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5457999Z frame #6: <unknown function> + 0x34b71 (0x7f6f88bc3b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5458294Z frame #7: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5458506Z frame #8: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5458667Z SIGABRT(6), PID: 1811300, Thread 1811820: 
2025-07-17T11:07:26.5459135Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5459412Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5459686Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5459886Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5460383Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5460589Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5460801Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5460888Z SIGABRT(6), PID: 1811296, Thread 1811435: 
2025-07-17T11:07:26.5461357Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5461558Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5461766Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5461963Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5462459Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5462666Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5462875Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5462963Z SIGABRT(6), PID: 1811294, Thread 1811326: 
2025-07-17T11:07:26.5463432Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5463635Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5463838Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5464032Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5464528Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5464735Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5464946Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5464950Z 
2025-07-17T11:07:26.5464954Z 
2025-07-17T11:07:26.5464957Z 
2025-07-17T11:07:26.5464960Z 
2025-07-17T11:07:26.5465047Z SIGABRT(6), PID: 1811300, Thread 1811821: 
2025-07-17T11:07:26.5465516Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5465720Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5471979Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5472383Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5473010Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5473246Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5473528Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5473535Z 
2025-07-17T11:07:26.5473636Z SIGABRT(6), PID: 1811296, Thread 1811437: 
2025-07-17T11:07:26.5474201Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5474419Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5474630Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5474836Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5475347Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5475557Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5475768Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5475774Z 
2025-07-17T11:07:26.5475868Z SIGABRT(6), PID: 1811294, Thread 1811327: 
2025-07-17T11:07:26.5476349Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5476556Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5476767Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5476972Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5477477Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5477684Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5477893Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5477899Z 
2025-07-17T11:07:26.5477990Z SIGABRT(6), PID: 1811297, Thread 1816108: 
2025-07-17T11:07:26.5478469Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5478677Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5478844Z frame #2: __poll + 0x4f (0x7f710ca2bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5479061Z frame #3: <unknown function> + 0xd4540 (0x7f6fd0e3d540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5479264Z frame #4: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5479470Z frame #5: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5479475Z 
2025-07-17T11:07:26.5479564Z SIGABRT(6), PID: 1811300, Thread 1811822: 
2025-07-17T11:07:26.5480032Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5480239Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5480441Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5480826Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5481390Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5481662Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5481876Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5481943Z 
2025-07-17T11:07:26.5482033Z SIGABRT(6), PID: 1811297, Thread 1816119: 
2025-07-17T11:07:26.5482501Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5482704Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5482874Z frame #2: __poll + 0x4f (0x7f710ca2bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5483088Z frame #3: <unknown function> + 0xd65a2 (0x7f6fd0e3f5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5483294Z frame #4: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5483500Z frame #5: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5483506Z 
2025-07-17T11:07:26.5483593Z SIGABRT(6), PID: 1811296, Thread 1811439: 
2025-07-17T11:07:26.5484053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5484259Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5484459Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5484662Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5485158Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5485365Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5485570Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5485574Z 
2025-07-17T11:07:26.5485665Z SIGABRT(6), PID: 1811294, Thread 1811328: 
2025-07-17T11:07:26.5486124Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5486331Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5486535Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5486736Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5487229Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5487438Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5487643Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5487651Z 
2025-07-17T11:07:26.5487739Z SIGABRT(6), PID: 1811297, Thread 1816126: 
2025-07-17T11:07:26.5488200Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5488405Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5488712Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5488970Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5489194Z frame #4: <unknown function> + 0xd1e7f (0x7f6fd0e3ae7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5489459Z frame #5: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5489669Z frame #6: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5489731Z 
2025-07-17T11:07:26.5489821Z SIGABRT(6), PID: 1811300, Thread 1811823: 
2025-07-17T11:07:26.5490287Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5490490Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5490692Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5490890Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5491386Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5491593Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5491803Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5491807Z 
2025-07-17T11:07:26.5491893Z SIGABRT(6), PID: 1811296, Thread 1811441: 
2025-07-17T11:07:26.5492356Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5492562Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5492763Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5492957Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5493454Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5493657Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5493865Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5493870Z 
2025-07-17T11:07:26.5493954Z SIGABRT(6), PID: 1811294, Thread 1811329: 
2025-07-17T11:07:26.5494415Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5494621Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5494825Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5495019Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5495515Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5495722Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5496017Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5496021Z 
2025-07-17T11:07:26.5496110Z SIGABRT(6), PID: 1811297, Thread 1816135: 
2025-07-17T11:07:26.5496577Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5496942Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5497151Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5497353Z frame #3: <unknown function> + 0x9ca53 (0x7f710c9afa53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5497683Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55b2cbef0bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5497991Z frame #5: <unknown function> + 0x1d314f (0x55b2cbfaf14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5498218Z frame #6: <unknown function> + 0x1d2f6b (0x55b2cbfaef6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5498440Z frame #7: <unknown function> + 0x6df36 (0x55b2cbe49f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5498691Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5498913Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5499158Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5499384Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5499621Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5499845Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5500077Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5500297Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5500532Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5500751Z frame #17: <unknown function> + 0x147734 (0x55b2cbf23734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5500975Z frame #18: <unknown function> + 0x21ca82 (0x55b2cbff8a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5501196Z frame #19: <unknown function> + 0x21c9e4 (0x55b2cbff89e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5501414Z frame #20: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5501627Z frame #21: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5501721Z SIGABRT(6), PID: 1811300, Thread 1811824: 
2025-07-17T11:07:26.5502195Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5502409Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5502612Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5502815Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5503318Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5503526Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5503736Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5503827Z SIGABRT(6), PID: 1811296, Thread 1811443: 
2025-07-17T11:07:26.5504293Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5504576Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5504780Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5505042Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5505606Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5505817Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5506083Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5506089Z 
2025-07-17T11:07:26.5506177Z SIGABRT(6), PID: 1811299, Thread 1816171: 
2025-07-17T11:07:26.5506647Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5506855Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5507061Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5507253Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5507633Z frame #4: <unknown function> + 0x5a02023 (0x7f6d63d47023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5508196Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6d63d4b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5508791Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6d63d421d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5509173Z frame #7: <unknown function> + 0x79d655 (0x7f6d75232655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5509412Z frame #8: <unknown function> + 0xdc253 (0x7f6d2c27d253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5509625Z frame #9: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5509842Z frame #10: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5509849Z 
2025-07-17T11:07:26.5509852Z 
2025-07-17T11:07:26.5509855Z 
2025-07-17T11:07:26.5509945Z SIGABRT(6), PID: 1811294, Thread 1811330: 
2025-07-17T11:07:26.5510432Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5510644Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5510853Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5511056Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5511567Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5511774Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5511988Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5511993Z 
2025-07-17T11:07:26.5512081Z SIGABRT(6), PID: 1811300, Thread 1811825: 
2025-07-17T11:07:26.5512552Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5512835Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5513101Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5513304Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5513863Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5514073Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5514364Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5514453Z SIGABRT(6), PID: 1811296, Thread 1811445: 
2025-07-17T11:07:26.5514926Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5515166Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5515382Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5515591Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5516085Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5516295Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5516500Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5516504Z 
2025-07-17T11:07:26.5516508Z 
2025-07-17T11:07:26.5516598Z SIGABRT(6), PID: 1811297, Thread 1816148: 
2025-07-17T11:07:26.5517064Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5517271Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5517437Z frame #2: read + 0x4c (0x7f710ca2781c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5517620Z frame #3: _Py_read + 0x4c (0x55b2cbfb6a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5517852Z frame #4: <unknown function> + 0x208c3b (0x55b2cbfe4c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5518074Z frame #5: <unknown function> + 0x14648d (0x55b2cbf2248d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5518321Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55b2cbf31215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5518544Z frame #7: <unknown function> + 0x208ae7 (0x55b2cbfe4ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5518764Z frame #8: <unknown function> + 0x208472 (0x55b2cbfe4472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5518986Z frame #9: <unknown function> + 0x1da6ee (0x55b2cbfb66ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5519213Z frame #10: <unknown function> + 0x1483bc (0x55b2cbf243bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5519458Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5519701Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5519941Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55b2cbf083bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5520159Z frame #14: <unknown function> + 0x147734 (0x55b2cbf23734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5520405Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b2cbf0ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5520701Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5520935Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5521215Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55b2cbf180ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5521453Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55b2cbf087b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5521732Z frame #20: <unknown function> + 0x147734 (0x55b2cbf23734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5522027Z frame #21: <unknown function> + 0x21ca82 (0x55b2cbff8a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5522245Z frame #22: <unknown function> + 0x21c9e4 (0x55b2cbff89e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5522457Z frame #23: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5522668Z frame #24: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5522674Z 
2025-07-17T11:07:26.5522766Z SIGABRT(6), PID: 1811294, Thread 1811331: 
2025-07-17T11:07:26.5523240Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5523454Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5523660Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5523862Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5524364Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5524574Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5524784Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5524788Z 
2025-07-17T11:07:26.5524877Z SIGABRT(6), PID: 1811296, Thread 1811447: 
2025-07-17T11:07:26.5525344Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5525551Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5525751Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5525951Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5526449Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5526658Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5526863Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5526963Z SIGABRT(6), PID: 1811300, Thread 1811826: 
2025-07-17T11:07:26.5527429Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5527636Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5527838Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5528035Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5528527Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5528817Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5529085Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5529090Z 
2025-07-17T11:07:26.5529093Z 
2025-07-17T11:07:26.5529186Z SIGABRT(6), PID: 1811294, Thread 1811332: 
2025-07-17T11:07:26.5529712Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5529983Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5530188Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5530384Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5530879Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5531087Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5531300Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5531305Z 
2025-07-17T11:07:26.5531392Z SIGABRT(6), PID: 1811296, Thread 1811449: 
2025-07-17T11:07:26.5531871Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5532082Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5532284Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5532477Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5532981Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5533187Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5533396Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5533400Z 
2025-07-17T11:07:26.5533489Z SIGABRT(6), PID: 1811300, Thread 1811827: 
2025-07-17T11:07:26.5533954Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5534159Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5534360Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5534557Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5535051Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5535255Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5535467Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5535470Z 
2025-07-17T11:07:26.5535556Z SIGABRT(6), PID: 1811294, Thread 1811333: 
2025-07-17T11:07:26.5536097Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5536301Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5536506Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5536820Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5537395Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5537606Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5537888Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5537962Z 
2025-07-17T11:07:26.5538052Z SIGABRT(6), PID: 1811296, Thread 1811450: 
2025-07-17T11:07:26.5538525Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5538731Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5538939Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5539133Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5539633Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5539839Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5540045Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5540051Z 
2025-07-17T11:07:26.5540139Z SIGABRT(6), PID: 1811300, Thread 1811828: 
2025-07-17T11:07:26.5540607Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5540811Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5541014Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5541214Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5541710Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5541915Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5542125Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5542129Z 
2025-07-17T11:07:26.5542222Z SIGABRT(6), PID: 1811294, Thread 1811334: 
2025-07-17T11:07:26.5542686Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5542895Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5543102Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5543299Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5543798Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5544008Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5544214Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5544218Z 
2025-07-17T11:07:26.5544309Z SIGABRT(6), PID: 1811296, Thread 1811452: 
2025-07-17T11:07:26.5544779Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5545064Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5545328Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5545527Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5546100Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5546370Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5546575Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5546579Z 
2025-07-17T11:07:26.5546667Z SIGABRT(6), PID: 1811300, Thread 1811829: 
2025-07-17T11:07:26.5547130Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5547337Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5547540Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5547743Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5548233Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5548442Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5548647Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5548653Z 
2025-07-17T11:07:26.5548743Z SIGABRT(6), PID: 1811294, Thread 1811335: 
2025-07-17T11:07:26.5549209Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5549416Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5549616Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5549815Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5550309Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5550517Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5550723Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5550731Z 
2025-07-17T11:07:26.5550818Z SIGABRT(6), PID: 1811299, Thread 1816172: 
2025-07-17T11:07:26.5551287Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5551491Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5551692Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5551888Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5552265Z frame #4: <unknown function> + 0x5a02023 (0x7f6d63d47023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5552830Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6d63d4b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5553561Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6d63d421d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5553946Z frame #7: <unknown function> + 0x79d655 (0x7f6d75232655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5554239Z frame #8: <unknown function> + 0xdc253 (0x7f6d2c27d253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5554505Z frame #9: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5554727Z frame #10: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5554815Z SIGABRT(6), PID: 1811296, Thread 1811454: 
2025-07-17T11:07:26.5555287Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5555494Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5555699Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5555900Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5556402Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5556609Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5556817Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5556821Z 
2025-07-17T11:07:26.5556824Z 
2025-07-17T11:07:26.5556912Z SIGABRT(6), PID: 1811300, Thread 1811830: 
2025-07-17T11:07:26.5557382Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5557586Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5557788Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5557981Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5558476Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5558680Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5558890Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5558896Z 
2025-07-17T11:07:26.5558981Z SIGABRT(6), PID: 1811294, Thread 1811336: 
2025-07-17T11:07:26.5559447Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5559650Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5559855Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5560049Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5560545Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5560748Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5560957Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5561035Z 
2025-07-17T11:07:26.5561127Z SIGABRT(6), PID: 1811300, Thread 1811831: 
2025-07-17T11:07:26.5561656Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5561867Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5562125Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5562382Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5562877Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5563093Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5563306Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5563310Z 
2025-07-17T11:07:26.5563403Z SIGABRT(6), PID: 1811296, Thread 1811456: 
2025-07-17T11:07:26.5563867Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5564077Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5564279Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5564481Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5564979Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5565195Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5565399Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5565403Z 
2025-07-17T11:07:26.5565494Z SIGABRT(6), PID: 1811297, Thread 1816192: 
2025-07-17T11:07:26.5565960Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5566167Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5566367Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5566564Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5566934Z frame #4: <unknown function> + 0x5a02023 (0x7f7050dd3023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5567496Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7050dd71ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5568085Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7050dce1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5568467Z frame #7: <unknown function> + 0x79d655 (0x7f70622be655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5568695Z frame #8: <unknown function> + 0xdc253 (0x7f7019309253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5568902Z frame #9: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5569111Z frame #10: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5569186Z 
2025-07-17T11:07:26.5569282Z SIGABRT(6), PID: 1811294, Thread 1811337: 
2025-07-17T11:07:26.5569809Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5570025Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5570287Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5570488Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5571045Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5571253Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5571465Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5571471Z 
2025-07-17T11:07:26.5571559Z SIGABRT(6), PID: 1811296, Thread 1811458: 
2025-07-17T11:07:26.5572028Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5572230Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5572435Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5572631Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5573128Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5573331Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5573542Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5573546Z 
2025-07-17T11:07:26.5573633Z SIGABRT(6), PID: 1811300, Thread 1811832: 
2025-07-17T11:07:26.5574100Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5574306Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5574508Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5574706Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5575198Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5575407Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5575618Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5575624Z 
2025-07-17T11:07:26.5575710Z SIGABRT(6), PID: 1811294, Thread 1811338: 
2025-07-17T11:07:26.5576259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5576464Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5576670Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5576864Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5577369Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5577668Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5577977Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5577982Z 
2025-07-17T11:07:26.5578073Z SIGABRT(6), PID: 1811296, Thread 1811460: 
2025-07-17T11:07:26.5578617Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5578893Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5579100Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5579296Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5579791Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5579998Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5580205Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5580209Z 
2025-07-17T11:07:26.5580296Z SIGABRT(6), PID: 1811300, Thread 1811833: 
2025-07-17T11:07:26.5580764Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5580968Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5581169Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5581364Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5581859Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5582068Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5582277Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5582282Z 
2025-07-17T11:07:26.5582371Z SIGABRT(6), PID: 1811294, Thread 1811339: 
2025-07-17T11:07:26.5582838Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5583047Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5583248Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5583448Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5583940Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5584147Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5584357Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5584362Z 
2025-07-17T11:07:26.5584450Z SIGABRT(6), PID: 1811296, Thread 1811462: 
2025-07-17T11:07:26.5584916Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5585121Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5585321Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5585593Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5586147Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5586360Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5586624Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5586683Z 
2025-07-17T11:07:26.5586777Z SIGABRT(6), PID: 1811300, Thread 1811834: 
2025-07-17T11:07:26.5587261Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5587472Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5587676Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5587876Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5588375Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5588585Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5588791Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5588797Z 
2025-07-17T11:07:26.5588888Z SIGABRT(6), PID: 1811294, Thread 1811341: 
2025-07-17T11:07:26.5589352Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5589562Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5589763Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5589960Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5590463Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5590676Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5590885Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5590889Z 
2025-07-17T11:07:26.5590977Z SIGABRT(6), PID: 1811296, Thread 1811464: 
2025-07-17T11:07:26.5591438Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5591646Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5591851Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5592051Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5592543Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5592751Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5592960Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5592964Z 
2025-07-17T11:07:26.5593051Z SIGABRT(6), PID: 1811300, Thread 1811835: 
2025-07-17T11:07:26.5593519Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5593801Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5594072Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5594270Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5594830Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5595098Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5595310Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5595314Z 
2025-07-17T11:07:26.5595400Z SIGABRT(6), PID: 1811294, Thread 1811342: 
2025-07-17T11:07:26.5595868Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5596078Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5596283Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5596480Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5596985Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5597199Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5597410Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5597415Z 
2025-07-17T11:07:26.5597501Z SIGABRT(6), PID: 1811296, Thread 1811466: 
2025-07-17T11:07:26.5597975Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5598179Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5598384Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5598578Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5599075Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5599278Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5599488Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5599494Z 
2025-07-17T11:07:26.5599580Z SIGABRT(6), PID: 1811299, Thread 1816173: 
2025-07-17T11:07:26.5600049Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5600253Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5600460Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5600655Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5601028Z frame #4: <unknown function> + 0x5a02023 (0x7f6d63d47023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5601590Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6d63d4b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5602319Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6d63d421d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5602758Z frame #7: <unknown function> + 0x79d655 (0x7f6d75232655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5602993Z frame #8: <unknown function> + 0xdc253 (0x7f6d2c27d253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5603379Z frame #9: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5603593Z frame #10: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5603598Z 
2025-07-17T11:07:26.5603689Z SIGABRT(6), PID: 1811300, Thread 1811836: 
2025-07-17T11:07:26.5604158Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5604369Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5604570Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5604767Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5605268Z frame #4: <unknown function> + 0x3475ab (0x7f222a9c35ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5605477Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5605683Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5605687Z 
2025-07-17T11:07:26.5605779Z SIGABRT(6), PID: 1811296, Thread 1811468: 
2025-07-17T11:07:26.5606243Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5606451Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5606652Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5606850Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5607344Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5607551Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5607754Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5607760Z 
2025-07-17T11:07:26.5607848Z SIGABRT(6), PID: 1811294, Thread 1811344: 
2025-07-17T11:07:26.5608312Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5608519Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5608721Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5608918Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5609413Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5609622Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5609936Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5609940Z 
2025-07-17T11:07:26.5610031Z SIGABRT(6), PID: 1811300, Thread 1812358: 
2025-07-17T11:07:26.5610558Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5611069Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f213c22308a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5611278Z frame #2: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5611552Z frame #3: pthread_kill + 0x12c (0x7f222db2d9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5611716Z frame #4: raise + 0x16 (0x7f222dad9476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5611876Z frame #5: abort + 0xd3 (0x7f222dabf7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5612118Z frame #6: <unknown function> + 0x972cc (0x7f20a9dac2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5612346Z frame #7: <unknown function> + 0x95f47 (0x7f20a9daaf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5612569Z frame #8: <unknown function> + 0x34b71 (0x7f20a9d49b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5612780Z frame #9: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5612999Z frame #10: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5613005Z 
2025-07-17T11:07:26.5613097Z SIGABRT(6), PID: 1811297, Thread 1816193: 
2025-07-17T11:07:26.5613565Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5613771Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5613972Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5614173Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5614544Z frame #4: <unknown function> + 0x5a02023 (0x7f7050dd3023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5615113Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7050dd71ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5615704Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7050dce1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5616167Z frame #7: <unknown function> + 0x79d655 (0x7f70622be655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5616402Z frame #8: <unknown function> + 0xdc253 (0x7f7019309253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5616609Z frame #9: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5616822Z frame #10: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5616826Z 
2025-07-17T11:07:26.5616914Z SIGABRT(6), PID: 1811296, Thread 1811470: 
2025-07-17T11:07:26.5617387Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5617592Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5617794Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5617988Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5618580Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5618862Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5619077Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5619081Z 
2025-07-17T11:07:26.5619242Z SIGABRT(6), PID: 1811294, Thread 1811346: 
2025-07-17T11:07:26.5619716Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5619995Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5620198Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5620397Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5620899Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5621105Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5621317Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5621321Z 
2025-07-17T11:07:26.5621407Z SIGABRT(6), PID: 1811300, Thread 1815482: 
2025-07-17T11:07:26.5621876Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5622077Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5622245Z frame #2: __poll + 0x4f (0x7f222dbafbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5622655Z frame #3: <unknown function> + 0x489f (0x7f222d6bb89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.5622885Z frame #4: <unknown function> + 0x1483bc (0x5558ca0f13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5623127Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5623359Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5623596Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5623820Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5624062Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5558ca0d9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5624295Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5624541Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5558ca0d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5624769Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5625002Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5625225Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5625456Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5625675Z frame #16: <unknown function> + 0x147734 (0x5558ca0f0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5625891Z frame #17: <unknown function> + 0x21ca82 (0x5558ca1c5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5626110Z frame #18: <unknown function> + 0x21c9e4 (0x5558ca1c59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5626397Z frame #19: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5626668Z frame #20: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5626673Z 
2025-07-17T11:07:26.5626768Z SIGABRT(6), PID: 1811296, Thread 1811471: 
2025-07-17T11:07:26.5627311Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5627579Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5627787Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5627986Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5628487Z frame #4: <unknown function> + 0x3475ab (0x7f1e740765ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5628697Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5628904Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5628909Z 
2025-07-17T11:07:26.5629000Z SIGABRT(6), PID: 1811294, Thread 1811347: 
2025-07-17T11:07:26.5629472Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5629683Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5629884Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5630081Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5630579Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5630787Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5630994Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5630998Z 
2025-07-17T11:07:26.5631090Z SIGABRT(6), PID: 1811296, Thread 1812360: 
2025-07-17T11:07:26.5631551Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5632003Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f1d858d308a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5632206Z frame #2: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5632401Z frame #3: pthread_kill + 0x12c (0x7f1e771e09fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5632564Z frame #4: raise + 0x16 (0x7f1e7718c476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5632725Z frame #5: abort + 0xd3 (0x7f1e771727f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5632957Z frame #6: <unknown function> + 0x972cc (0x7f1cf345c2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5633190Z frame #7: <unknown function> + 0x95f47 (0x7f1cf345af47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5633419Z frame #8: <unknown function> + 0x34b71 (0x7f1cf33f9b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5633628Z frame #9: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5633841Z frame #10: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5633915Z 
2025-07-17T11:07:26.5634009Z SIGABRT(6), PID: 1811294, Thread 1811349: 
2025-07-17T11:07:26.5634539Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5634754Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5634959Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5635218Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5635794Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5636002Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5636210Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5636217Z 
2025-07-17T11:07:26.5636309Z SIGABRT(6), PID: 1811296, Thread 1815280: 
2025-07-17T11:07:26.5636778Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5636984Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5637147Z frame #2: __poll + 0x4f (0x7f1e77262bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5637552Z frame #3: <unknown function> + 0x489f (0x7f1e76d6e89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.5637776Z frame #4: <unknown function> + 0x1483bc (0x560c8d9f13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5638020Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5638245Z frame #6: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5638481Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5638703Z frame #8: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5638946Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x560c8d9d9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5639172Z frame #10: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5639421Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x560c8d9d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5639641Z frame #12: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5639875Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5640096Z frame #14: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5640329Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5640547Z frame #16: <unknown function> + 0x147734 (0x560c8d9f0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5640768Z frame #17: <unknown function> + 0x21ca82 (0x560c8dac5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5640985Z frame #18: <unknown function> + 0x21c9e4 (0x560c8dac59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5641196Z frame #19: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5641407Z frame #20: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5641412Z 
2025-07-17T11:07:26.5641505Z SIGABRT(6), PID: 1811294, Thread 1811352: 
2025-07-17T11:07:26.5641973Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5642261Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5642524Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5642730Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5643286Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5643557Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5643766Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5643856Z SIGABRT(6), PID: 1811300, Thread 1815483: 
2025-07-17T11:07:26.5644324Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5644537Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5644738Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5644960Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f222db2b2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5645399Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f2167f6b048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5645820Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f2167f6dbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5646048Z frame #6: <unknown function> + 0xdc253 (0x7f213a48f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5646254Z frame #7: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5646464Z frame #8: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5646468Z 
2025-07-17T11:07:26.5646473Z 
2025-07-17T11:07:26.5646560Z SIGABRT(6), PID: 1811299, Thread 1816174: 
2025-07-17T11:07:26.5647042Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5647249Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5647453Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5647648Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5648023Z frame #4: <unknown function> + 0x5a02023 (0x7f6d63d47023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5648591Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6d63d4b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5649188Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6d63d421d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5649564Z frame #7: <unknown function> + 0x79d655 (0x7f6d75232655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5649793Z frame #8: <unknown function> + 0xdc253 (0x7f6d2c27d253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5649997Z frame #9: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5650212Z frame #10: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5650286Z 
2025-07-17T11:07:26.5650376Z SIGABRT(6), PID: 1811294, Thread 1811353: 
2025-07-17T11:07:26.5650907Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5651114Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5651376Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5651579Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5652147Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5652353Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5652565Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5652570Z 
2025-07-17T11:07:26.5652659Z SIGABRT(6), PID: 1811300, Thread 1815484: 
2025-07-17T11:07:26.5653134Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5653341Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5653544Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5653764Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f222db2b2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5654239Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f2167f65741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5654462Z frame #5: <unknown function> + 0xdc253 (0x7f213a48f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5654669Z frame #6: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5654875Z frame #7: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5654881Z 
2025-07-17T11:07:26.5654971Z SIGABRT(6), PID: 1811294, Thread 1811355: 
2025-07-17T11:07:26.5655437Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5655643Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5655848Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5656113Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5656613Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5656823Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5657030Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5657034Z 
2025-07-17T11:07:26.5657119Z SIGABRT(6), PID: 1811300, Thread 1815485: 
2025-07-17T11:07:26.5657586Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5657792Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5657960Z frame #2: ioctl + 0x3f (0x7f222dbb194f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5658193Z frame #3: <unknown function> + 0x13eef0 (0x7f20a9e53ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5658518Z frame #4: <unknown function> + 0x1378c4 (0x7f20a9e4c8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5658834Z frame #5: <unknown function> + 0x95b2b (0x7f20a9daab2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5659080Z frame #6: <unknown function> + 0x34b71 (0x7f20a9d49b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5659296Z frame #7: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5659580Z frame #8: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5659743Z SIGABRT(6), PID: 1811297, Thread 1816194: 
2025-07-17T11:07:26.5660236Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5660449Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5660659Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5660861Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5661247Z frame #4: <unknown function> + 0x5a02023 (0x7f7050dd3023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5661816Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7050dd71ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5662414Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7050dce1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5662794Z frame #7: <unknown function> + 0x79d655 (0x7f70622be655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5663024Z frame #8: <unknown function> + 0xdc253 (0x7f7019309253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5663231Z frame #9: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5663447Z frame #10: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5663534Z SIGABRT(6), PID: 1811296, Thread 1815281: 
2025-07-17T11:07:26.5664010Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5664218Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5664423Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5664643Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f1e771de2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5665083Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f1db161b048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5665499Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f1db161dbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5665732Z frame #6: <unknown function> + 0xdc253 (0x7f1d83b3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5665939Z frame #7: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5666150Z frame #8: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5666238Z SIGABRT(6), PID: 1811294, Thread 1811357: 
2025-07-17T11:07:26.5666707Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5666987Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5667193Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5667455Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5668055Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5668265Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5668537Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5668541Z 
2025-07-17T11:07:26.5668545Z 
2025-07-17T11:07:26.5668548Z 
2025-07-17T11:07:26.5668551Z 
2025-07-17T11:07:26.5668641Z SIGABRT(6), PID: 1811294, Thread 1811358: 
2025-07-17T11:07:26.5669109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5669323Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5669535Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5669741Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5670257Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5670471Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5670680Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5670684Z 
2025-07-17T11:07:26.5670776Z SIGABRT(6), PID: 1811300, Thread 1816109: 
2025-07-17T11:07:26.5671251Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5671464Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5671632Z frame #2: __poll + 0x4f (0x7f222dbafbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5671849Z frame #3: <unknown function> + 0xd4540 (0x7f20f1fc3540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5672053Z frame #4: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5672263Z frame #5: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5672267Z 
2025-07-17T11:07:26.5672354Z SIGABRT(6), PID: 1811296, Thread 1815282: 
2025-07-17T11:07:26.5672821Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5673028Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5673231Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5673455Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f1e771de2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5673937Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f1db1615741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5674171Z frame #5: <unknown function> + 0xdc253 (0x7f1d83b3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5674378Z frame #6: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5674585Z frame #7: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5674661Z 
2025-07-17T11:07:26.5674759Z SIGABRT(6), PID: 1811294, Thread 1811361: 
2025-07-17T11:07:26.5675284Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5675501Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5675764Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5675971Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5676535Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5676745Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5676950Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5677042Z SIGABRT(6), PID: 1811300, Thread 1816118: 
2025-07-17T11:07:26.5677518Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5677726Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5677891Z frame #2: __poll + 0x4f (0x7f222dbafbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5678104Z frame #3: <unknown function> + 0xd65a2 (0x7f20f1fc55a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5678310Z frame #4: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5678518Z frame #5: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5678522Z 
2025-07-17T11:07:26.5678526Z 
2025-07-17T11:07:26.5678618Z SIGABRT(6), PID: 1811296, Thread 1815285: 
2025-07-17T11:07:26.5679081Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5679289Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5679452Z frame #2: ioctl + 0x3f (0x7f1e7726494f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5679692Z frame #3: <unknown function> + 0x13eef0 (0x7f1cf3503ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5679922Z frame #4: <unknown function> + 0x1378c4 (0x7f1cf34fc8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5680152Z frame #5: <unknown function> + 0x95b2b (0x7f1cf345ab2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5680377Z frame #6: <unknown function> + 0x34b71 (0x7f1cf33f9b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5680586Z frame #7: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5680796Z frame #8: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5680800Z 
2025-07-17T11:07:26.5680889Z SIGABRT(6), PID: 1811299, Thread 1816175: 
2025-07-17T11:07:26.5681354Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6d2e010b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5681563Z frame #1: <unknown function> + 0x42520 (0x7f6e1f8c7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5681764Z frame #2: <unknown function> + 0x91117 (0x7f6e1f916117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5681968Z frame #3: pthread_cond_wait + 0x211 (0x7f6e1f918a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5682347Z frame #4: <unknown function> + 0x5a02023 (0x7f6d63d47023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5682988Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6d63d4b1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5683633Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6d63d421d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5684074Z frame #7: <unknown function> + 0x79d655 (0x7f6d75232655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5684364Z frame #8: <unknown function> + 0xdc253 (0x7f6d2c27d253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5684575Z frame #9: <unknown function> + 0x94ac3 (0x7f6e1f919ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5684789Z frame #10: <unknown function> + 0x126850 (0x7f6e1f9ab850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5684796Z 
2025-07-17T11:07:26.5684888Z SIGABRT(6), PID: 1811300, Thread 1816125: 
2025-07-17T11:07:26.5685368Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5685578Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5685781Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5685984Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5686202Z frame #4: <unknown function> + 0xd1e7f (0x7f20f1fc0e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5686407Z frame #5: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5686612Z frame #6: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5686615Z 
2025-07-17T11:07:26.5686708Z SIGABRT(6), PID: 1811294, Thread 1811363: 
2025-07-17T11:07:26.5687171Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5687380Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5687581Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5687783Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5688287Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5688496Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5688704Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5688709Z 
2025-07-17T11:07:26.5688798Z SIGABRT(6), PID: 1811296, Thread 1816111: 
2025-07-17T11:07:26.5689263Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5689469Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5689634Z frame #2: __poll + 0x4f (0x7f1e77262bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5689847Z frame #3: <unknown function> + 0xd4540 (0x7f1d3b673540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5690053Z frame #4: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5690258Z frame #5: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5690262Z 
2025-07-17T11:07:26.5690348Z SIGABRT(6), PID: 1811296, Thread 1816117: 
2025-07-17T11:07:26.5690890Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5691156Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5691319Z frame #2: __poll + 0x4f (0x7f1e77262bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5691608Z frame #3: <unknown function> + 0xd65a2 (0x7f1d3b6755a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5691818Z frame #4: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5692092Z frame #5: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5692181Z SIGABRT(6), PID: 1811294, Thread 1811365: 
2025-07-17T11:07:26.5692646Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5692854Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5693064Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5693264Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5693769Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5693975Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5694185Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5694189Z 
2025-07-17T11:07:26.5694193Z 
2025-07-17T11:07:26.5694278Z SIGABRT(6), PID: 1811300, Thread 1816132: 
2025-07-17T11:07:26.5694751Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5694955Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5695162Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5695363Z frame #3: <unknown function> + 0x9ca53 (0x7f222db33a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5695620Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5558ca0bdbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5695852Z frame #5: <unknown function> + 0x1d314f (0x5558ca17c14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5696136Z frame #6: <unknown function> + 0x1d2f6b (0x5558ca17bf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5696356Z frame #7: <unknown function> + 0x6df36 (0x5558ca016f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5696606Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5696829Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5697072Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5697297Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5697532Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5697757Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5697989Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5698210Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5698442Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5698750Z frame #17: <unknown function> + 0x147734 (0x5558ca0f0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5699066Z frame #18: <unknown function> + 0x21ca82 (0x5558ca1c5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5699288Z frame #19: <unknown function> + 0x21c9e4 (0x5558ca1c59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5699569Z frame #20: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5699785Z frame #21: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5699860Z 
2025-07-17T11:07:26.5699956Z SIGABRT(6), PID: 1811296, Thread 1816120: 
2025-07-17T11:07:26.5700429Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5700641Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5700844Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5701046Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5701259Z frame #4: <unknown function> + 0xd1e7f (0x7f1d3b670e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5701467Z frame #5: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5701672Z frame #6: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5701678Z 
2025-07-17T11:07:26.5701766Z SIGABRT(6), PID: 1811294, Thread 1811367: 
2025-07-17T11:07:26.5702230Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5702439Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5702640Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5702841Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5703345Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5703559Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5703769Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5703773Z 
2025-07-17T11:07:26.5703863Z SIGABRT(6), PID: 1811300, Thread 1816145: 
2025-07-17T11:07:26.5704326Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5704534Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5704698Z frame #2: read + 0x4c (0x7f222dbab81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5704879Z frame #3: _Py_read + 0x4c (0x5558ca183a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5705102Z frame #4: <unknown function> + 0x208c3b (0x5558ca1b1c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5705326Z frame #5: <unknown function> + 0x14648d (0x5558ca0ef48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5705569Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5558ca0fe215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5705788Z frame #7: <unknown function> + 0x208ae7 (0x5558ca1b1ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5706005Z frame #8: <unknown function> + 0x208472 (0x5558ca1b1472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5706301Z frame #9: <unknown function> + 0x1da6ee (0x5558ca1836ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5706525Z frame #10: <unknown function> + 0x1483bc (0x5558ca0f13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5706824Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5707052Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5707355Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5558ca0d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5707630Z frame #14: <unknown function> + 0x147734 (0x5558ca0f0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5707885Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5558ca0d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5708106Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5708342Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5708565Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5558ca0e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5708801Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5558ca0d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5709017Z frame #20: <unknown function> + 0x147734 (0x5558ca0f0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5709238Z frame #21: <unknown function> + 0x21ca82 (0x5558ca1c5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5709454Z frame #22: <unknown function> + 0x21c9e4 (0x5558ca1c59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5709667Z frame #23: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5709875Z frame #24: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5709879Z 
2025-07-17T11:07:26.5709971Z SIGABRT(6), PID: 1811296, Thread 1816131: 
2025-07-17T11:07:26.5710440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5710649Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5710850Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5711058Z frame #3: <unknown function> + 0x9ca53 (0x7f1e771e6a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5711306Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x560c8d9bdbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5711531Z frame #5: <unknown function> + 0x1d314f (0x560c8da7c14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5711748Z frame #6: <unknown function> + 0x1d2f6b (0x560c8da7bf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5711968Z frame #7: <unknown function> + 0x6df36 (0x560c8d916f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5712210Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5712434Z frame #9: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5712668Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5712895Z frame #11: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5713127Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5713351Z frame #13: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5713580Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5713803Z frame #15: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5714113Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5714395Z frame #17: <unknown function> + 0x147734 (0x560c8d9f0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5714616Z frame #18: <unknown function> + 0x21ca82 (0x560c8dac5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5714893Z frame #19: <unknown function> + 0x21c9e4 (0x560c8dac59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5715105Z frame #20: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5715382Z frame #21: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5715387Z 
2025-07-17T11:07:26.5715475Z SIGABRT(6), PID: 1811297, Thread 1816195: 
2025-07-17T11:07:26.5715951Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5716162Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5716369Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5716568Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5716948Z frame #4: <unknown function> + 0x5a02023 (0x7f7050dd3023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5717520Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7050dd71ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5718104Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7050dce1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5718488Z frame #7: <unknown function> + 0x79d655 (0x7f70622be655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5718717Z frame #8: <unknown function> + 0xdc253 (0x7f7019309253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5718924Z frame #9: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5719136Z frame #10: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5719142Z 
2025-07-17T11:07:26.5719233Z SIGABRT(6), PID: 1811294, Thread 1811369: 
2025-07-17T11:07:26.5719700Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5719907Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5720111Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5720308Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5720814Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5721023Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5721236Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5721242Z 
2025-07-17T11:07:26.5721336Z SIGABRT(6), PID: 1811296, Thread 1816143: 
2025-07-17T11:07:26.5721804Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5722010Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5722247Z frame #2: read + 0x4c (0x7f1e7725e81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5722489Z frame #3: _Py_read + 0x4c (0x560c8da83a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5722717Z frame #4: <unknown function> + 0x208c3b (0x560c8dab1c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5723003Z frame #5: <unknown function> + 0x14648d (0x560c8d9ef48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5723249Z frame #6: PyObject_VectorcallMethod + 0x85 (0x560c8d9fe215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5723534Z frame #7: <unknown function> + 0x208ae7 (0x560c8dab1ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5723753Z frame #8: <unknown function> + 0x208472 (0x560c8dab1472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5723973Z frame #9: <unknown function> + 0x1da6ee (0x560c8da836ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5724197Z frame #10: <unknown function> + 0x1483bc (0x560c8d9f13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5724436Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5724661Z frame #12: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5724897Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x560c8d9d53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5725116Z frame #14: <unknown function> + 0x147734 (0x560c8d9f0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5725367Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x560c8d9d7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5725589Z frame #16: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5725823Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5726045Z frame #18: _PyFunction_Vectorcall + 0x6c (0x560c8d9e50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5726281Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x560c8d9d57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5726497Z frame #20: <unknown function> + 0x147734 (0x560c8d9f0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5726719Z frame #21: <unknown function> + 0x21ca82 (0x560c8dac5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5726947Z frame #22: <unknown function> + 0x21c9e4 (0x560c8dac59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5727163Z frame #23: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5727378Z frame #24: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5727382Z 
2025-07-17T11:07:26.5727471Z SIGABRT(6), PID: 1811294, Thread 1811371: 
2025-07-17T11:07:26.5727937Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5728152Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5728359Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5728560Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5729060Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5729270Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5729478Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5729482Z 
2025-07-17T11:07:26.5729573Z SIGABRT(6), PID: 1811294, Thread 1811373: 
2025-07-17T11:07:26.5730136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5730409Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5730619Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5730998Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5731504Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5731780Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5731988Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5731994Z 
2025-07-17T11:07:26.5732084Z SIGABRT(6), PID: 1811294, Thread 1811375: 
2025-07-17T11:07:26.5732556Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5732763Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5732969Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5733165Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5733665Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5733870Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5734077Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5734083Z 
2025-07-17T11:07:26.5734169Z SIGABRT(6), PID: 1811294, Thread 1811378: 
2025-07-17T11:07:26.5734645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5734855Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5735064Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5735261Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5735761Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5736036Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5736252Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5736257Z 
2025-07-17T11:07:26.5736342Z SIGABRT(6), PID: 1811294, Thread 1811381: 
2025-07-17T11:07:26.5736811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5737018Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5737224Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5737420Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5737915Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5738223Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5738433Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5738437Z 
2025-07-17T11:07:26.5738597Z SIGABRT(6), PID: 1811300, Thread 1816183: 
2025-07-17T11:07:26.5739072Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5739352Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5739631Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5739830Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5740207Z frame #4: <unknown function> + 0x5a02023 (0x7f2171f59023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5740775Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2171f5d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5741367Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2171f541d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5741745Z frame #7: <unknown function> + 0x79d655 (0x7f2183444655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5741980Z frame #8: <unknown function> + 0xdc253 (0x7f213a48f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5742185Z frame #9: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5742403Z frame #10: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5742409Z 
2025-07-17T11:07:26.5742498Z SIGABRT(6), PID: 1811294, Thread 1811384: 
2025-07-17T11:07:26.5742968Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5743175Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5743380Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5743579Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5744082Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5744288Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5744498Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5744502Z 
2025-07-17T11:07:26.5744592Z SIGABRT(6), PID: 1811297, Thread 1816196: 
2025-07-17T11:07:26.5745057Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5745262Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5745465Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5745662Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5746030Z frame #4: <unknown function> + 0x5a02023 (0x7f7050dd3023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5746588Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7050dd71ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5747314Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7050dce1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5747762Z frame #7: <unknown function> + 0x79d655 (0x7f70622be655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5747991Z frame #8: <unknown function> + 0xdc253 (0x7f7019309253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5748268Z frame #9: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5748481Z frame #10: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5748485Z 
2025-07-17T11:07:26.5748576Z SIGABRT(6), PID: 1811296, Thread 1816176: 
2025-07-17T11:07:26.5749044Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5749257Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5749460Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5749665Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5750031Z frame #4: <unknown function> + 0x5a02023 (0x7f1dbb609023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5750591Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1dbb60d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5751180Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1dbb6041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5751564Z frame #7: <unknown function> + 0x79d655 (0x7f1dccaf4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5751789Z frame #8: <unknown function> + 0xdc253 (0x7f1d83b3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5752000Z frame #9: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5752208Z frame #10: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5752217Z 
2025-07-17T11:07:26.5752304Z SIGABRT(6), PID: 1811294, Thread 1811387: 
2025-07-17T11:07:26.5752775Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5752983Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5753190Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5753387Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5753891Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5754095Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5754310Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5754314Z 
2025-07-17T11:07:26.5754400Z SIGABRT(6), PID: 1811294, Thread 1811388: 
2025-07-17T11:07:26.5754868Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5755153Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5755418Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5755619Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5756175Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5756460Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5756673Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5756677Z 
2025-07-17T11:07:26.5756763Z SIGABRT(6), PID: 1811294, Thread 1811393: 
2025-07-17T11:07:26.5757236Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5757449Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5757652Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5757844Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5758340Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5758546Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5758754Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5758758Z 
2025-07-17T11:07:26.5758845Z SIGABRT(6), PID: 1811294, Thread 1811396: 
2025-07-17T11:07:26.5759309Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5759514Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5759719Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5759916Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5760415Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5760624Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5760835Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5760841Z 
2025-07-17T11:07:26.5760928Z SIGABRT(6), PID: 1811300, Thread 1816184: 
2025-07-17T11:07:26.5761397Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5761599Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5761804Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5761999Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5762372Z frame #4: <unknown function> + 0x5a02023 (0x7f2171f59023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5762929Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2171f5d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5763647Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2171f541d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5764028Z frame #7: <unknown function> + 0x79d655 (0x7f2183444655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5764315Z frame #8: <unknown function> + 0xdc253 (0x7f213a48f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5764528Z frame #9: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5764802Z frame #10: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5764806Z 
2025-07-17T11:07:26.5764902Z SIGABRT(6), PID: 1811294, Thread 1811399: 
2025-07-17T11:07:26.5765374Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5765585Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5765790Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5765988Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5766489Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5766696Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5766906Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5766910Z 
2025-07-17T11:07:26.5766999Z SIGABRT(6), PID: 1811297, Thread 1816197: 
2025-07-17T11:07:26.5767473Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5767679Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5767882Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5768082Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5768453Z frame #4: <unknown function> + 0x5a02023 (0x7f7050dd3023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5769012Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7050dd71ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5769597Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7050dce1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5769977Z frame #7: <unknown function> + 0x79d655 (0x7f70622be655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5770201Z frame #8: <unknown function> + 0xdc253 (0x7f7019309253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5770411Z frame #9: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5770626Z frame #10: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5770632Z 
2025-07-17T11:07:26.5770721Z SIGABRT(6), PID: 1811296, Thread 1816177: 
2025-07-17T11:07:26.5771188Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5771397Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5771674Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5771950Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5772322Z frame #4: <unknown function> + 0x5a02023 (0x7f1dbb609023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5772947Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1dbb60d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5773593Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1dbb6041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5773972Z frame #7: <unknown function> + 0x79d655 (0x7f1dccaf4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5774204Z frame #8: <unknown function> + 0xdc253 (0x7f1d83b3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5774412Z frame #9: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5774627Z frame #10: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5774631Z 
2025-07-17T11:07:26.5774720Z SIGABRT(6), PID: 1811294, Thread 1811402: 
2025-07-17T11:07:26.5775192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5775402Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5775606Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5775807Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5776378Z frame #4: <unknown function> + 0x3475ab (0x7f7bd3da75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5776585Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5776796Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5776800Z 
2025-07-17T11:07:26.5776886Z SIGABRT(6), PID: 1811294, Thread 1812352: 
2025-07-17T11:07:26.5777356Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5777801Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f7ae560608a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5778009Z frame #2: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5778197Z frame #3: pthread_kill + 0x12c (0x7f7bd6f119fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5778365Z frame #4: raise + 0x16 (0x7f7bd6ebd476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5778523Z frame #5: abort + 0xd3 (0x7f7bd6ea37f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5778760Z frame #6: <unknown function> + 0x972cc (0x7f7a5318f2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5778986Z frame #7: <unknown function> + 0x95f47 (0x7f7a5318df47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5779212Z frame #8: <unknown function> + 0x34b71 (0x7f7a5312cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5779418Z frame #9: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5779635Z frame #10: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5779724Z 
2025-07-17T11:07:26.5779815Z SIGABRT(6), PID: 1811294, Thread 1814676: 
2025-07-17T11:07:26.5780368Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5780581Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5780822Z frame #2: __poll + 0x4f (0x7f7bd6f93bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5781229Z frame #3: <unknown function> + 0x489f (0x7f7bd6a9f89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.5781527Z frame #4: <unknown function> + 0x1483bc (0x56060cdb23bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5781771Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5781998Z frame #6: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5782229Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5782454Z frame #8: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5782693Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x56060cd9aa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5782925Z frame #10: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5783171Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x56060cd98c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5783395Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5783630Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5783856Z frame #14: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5784089Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5784310Z frame #16: <unknown function> + 0x147734 (0x56060cdb1734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5784527Z frame #17: <unknown function> + 0x21ca82 (0x56060ce86a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5784750Z frame #18: <unknown function> + 0x21c9e4 (0x56060ce869e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5784961Z frame #19: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5785176Z frame #20: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5785180Z 
2025-07-17T11:07:26.5785267Z SIGABRT(6), PID: 1811297, Thread 1816198: 
2025-07-17T11:07:26.5785736Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5785945Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5786153Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5786350Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5786727Z frame #4: <unknown function> + 0x5a02023 (0x7f7050dd3023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5787290Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7050dd71ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5787887Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7050dce1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5788343Z frame #7: <unknown function> + 0x79d655 (0x7f70622be655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5788628Z frame #8: <unknown function> + 0xdc253 (0x7f7019309253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5788843Z frame #9: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5789139Z frame #10: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5789198Z 
2025-07-17T11:07:26.5789292Z SIGABRT(6), PID: 1811300, Thread 1816186: 
2025-07-17T11:07:26.5789759Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5789969Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5790172Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5790370Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5790735Z frame #4: <unknown function> + 0x5a02023 (0x7f2171f59023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5791297Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2171f5d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5791883Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2171f541d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5792259Z frame #7: <unknown function> + 0x79d655 (0x7f2183444655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5792487Z frame #8: <unknown function> + 0xdc253 (0x7f213a48f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5792698Z frame #9: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5792911Z frame #10: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5792915Z 
2025-07-17T11:07:26.5793005Z SIGABRT(6), PID: 1811296, Thread 1816178: 
2025-07-17T11:07:26.5793472Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5793683Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5793881Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5794079Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5794445Z frame #4: <unknown function> + 0x5a02023 (0x7f1dbb609023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5795008Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1dbb60d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5795594Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1dbb6041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5795976Z frame #7: <unknown function> + 0x79d655 (0x7f1dccaf4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5796209Z frame #8: <unknown function> + 0xdc253 (0x7f1d83b3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5796414Z frame #9: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5796704Z frame #10: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5796711Z 
2025-07-17T11:07:26.5796863Z SIGABRT(6), PID: 1811294, Thread 1814677: 
2025-07-17T11:07:26.5797344Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5797610Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5797853Z frame #2: epoll_wait + 0x5e (0x7f7bd6fa0e2e in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5798181Z frame #3: uv__io_poll + 0x6bd (0x7f7b27b5c44d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5798495Z frame #4: uv_run + 0x12a (0x7f7b27b5036a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5798870Z frame #5: <unknown function> + 0x658a08a (0x7f7b1bec408a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5799100Z frame #6: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5799309Z frame #7: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5799522Z frame #8: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5799526Z 
2025-07-17T11:07:26.5799615Z SIGABRT(6), PID: 1811294, Thread 1816091: 
2025-07-17T11:07:26.5800084Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5800290Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5800495Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5800719Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f7bd6f0f2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5801158Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f7b1134e048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5801574Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f7b11350bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5801803Z frame #6: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5802007Z frame #7: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5802222Z frame #8: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5802226Z 
2025-07-17T11:07:26.5802312Z SIGABRT(6), PID: 1811297, Thread 1816199: 
2025-07-17T11:07:26.5802793Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f701b09cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5802998Z frame #1: <unknown function> + 0x42520 (0x7f710c955520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5803199Z frame #2: <unknown function> + 0x91117 (0x7f710c9a4117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5803397Z frame #3: pthread_cond_wait + 0x211 (0x7f710c9a6a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5803769Z frame #4: <unknown function> + 0x5a02023 (0x7f7050dd3023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5804326Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7050dd71ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5804917Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7050dce1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5805422Z frame #7: <unknown function> + 0x79d655 (0x7f70622be655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5805661Z frame #8: <unknown function> + 0xdc253 (0x7f7019309253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5805925Z frame #9: <unknown function> + 0x94ac3 (0x7f710c9a7ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5806195Z frame #10: <unknown function> + 0x126850 (0x7f710ca39850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5806199Z 
2025-07-17T11:07:26.5806288Z SIGABRT(6), PID: 1811300, Thread 1816187: 
2025-07-17T11:07:26.5806754Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5806963Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5807167Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5807369Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5807739Z frame #4: <unknown function> + 0x5a02023 (0x7f2171f59023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5808305Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2171f5d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5808885Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2171f541d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5809262Z frame #7: <unknown function> + 0x79d655 (0x7f2183444655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5809489Z frame #8: <unknown function> + 0xdc253 (0x7f213a48f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5809699Z frame #9: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5809908Z frame #10: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5809913Z 
2025-07-17T11:07:26.5810002Z SIGABRT(6), PID: 1811294, Thread 1816092: 
2025-07-17T11:07:26.5810466Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5810677Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5810880Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5811106Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f7bd6f0f2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5811573Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f7b11348741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.5811802Z frame #5: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5812009Z frame #6: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5812222Z frame #7: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5812226Z 
2025-07-17T11:07:26.5812313Z SIGABRT(6), PID: 1811296, Thread 1816179: 
2025-07-17T11:07:26.5812781Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5813058Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5813267Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5813523Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5813898Z frame #4: <unknown function> + 0x5a02023 (0x7f1dbb609023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5814514Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1dbb60d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5815165Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1dbb6041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5815546Z frame #7: <unknown function> + 0x79d655 (0x7f1dccaf4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5815776Z frame #8: <unknown function> + 0xdc253 (0x7f1d83b3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5816042Z frame #9: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5816255Z frame #10: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5816262Z 
2025-07-17T11:07:26.5816347Z SIGABRT(6), PID: 1811294, Thread 1816094: 
2025-07-17T11:07:26.5816842Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5817064Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5817233Z frame #2: ioctl + 0x3f (0x7f7bd6f9594f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5817478Z frame #3: <unknown function> + 0x13eef0 (0x7f7a53236ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5817706Z frame #4: <unknown function> + 0x1378c4 (0x7f7a5322f8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5817941Z frame #5: <unknown function> + 0x95b2b (0x7f7a5318db2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5818163Z frame #6: <unknown function> + 0x34b71 (0x7f7a5312cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5818375Z frame #7: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5818588Z frame #8: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5818593Z 
2025-07-17T11:07:26.5818685Z SIGABRT(6), PID: 1811294, Thread 1816105: 
2025-07-17T11:07:26.5819162Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5819378Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5819541Z frame #2: __poll + 0x4f (0x7f7bd6f93bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5819764Z frame #3: <unknown function> + 0xd4540 (0x7f7a9b3a6540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5819969Z frame #4: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5820178Z frame #5: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5820184Z 
2025-07-17T11:07:26.5820272Z SIGABRT(6), PID: 1811294, Thread 1816113: 
2025-07-17T11:07:26.5820745Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5820949Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5821210Z frame #2: __poll + 0x4f (0x7f7bd6f93bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5821424Z frame #3: <unknown function> + 0xd65a2 (0x7f7a9b3a85a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5821734Z frame #4: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5821946Z frame #5: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5821950Z 
2025-07-17T11:07:26.5822112Z SIGABRT(6), PID: 1811294, Thread 1816127: 
2025-07-17T11:07:26.5822580Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5822863Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5823068Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5823278Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5823491Z frame #4: <unknown function> + 0xd1e7f (0x7f7a9b3a3e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.5823699Z frame #5: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5823905Z frame #6: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5823909Z 
2025-07-17T11:07:26.5824001Z SIGABRT(6), PID: 1811294, Thread 1816128: 
2025-07-17T11:07:26.5824468Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5824673Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5824874Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5825082Z frame #3: <unknown function> + 0x9ca53 (0x7f7bd6f17a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5825333Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x56060cd7ebc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5825561Z frame #5: <unknown function> + 0x1d314f (0x56060ce3d14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5825781Z frame #6: <unknown function> + 0x1d2f6b (0x56060ce3cf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5826003Z frame #7: <unknown function> + 0x6df36 (0x56060ccd7f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5826241Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5826473Z frame #9: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5826712Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5826938Z frame #11: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5827172Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5827401Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5827630Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5827853Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5828092Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5828314Z frame #17: <unknown function> + 0x147734 (0x56060cdb1734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5828529Z frame #18: <unknown function> + 0x21ca82 (0x56060ce86a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5828748Z frame #19: <unknown function> + 0x21c9e4 (0x56060ce869e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5829035Z frame #20: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5829309Z frame #21: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5829313Z 
2025-07-17T11:07:26.5829404Z SIGABRT(6), PID: 1811300, Thread 1816188: 
2025-07-17T11:07:26.5829947Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5830213Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5830421Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5830621Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5830999Z frame #4: <unknown function> + 0x5a02023 (0x7f2171f59023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5831570Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2171f5d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5832165Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2171f541d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5832542Z frame #7: <unknown function> + 0x79d655 (0x7f2183444655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5832777Z frame #8: <unknown function> + 0xdc253 (0x7f213a48f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5832982Z frame #9: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5833198Z frame #10: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5833287Z SIGABRT(6), PID: 1811294, Thread 1816137: 
2025-07-17T11:07:26.5833761Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5833967Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5834136Z frame #2: read + 0x4c (0x7f7bd6f8f81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5834312Z frame #3: _Py_read + 0x4c (0x56060ce44a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5834539Z frame #4: <unknown function> + 0x208c3b (0x56060ce72c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5834756Z frame #5: <unknown function> + 0x14648d (0x56060cdb048d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5834998Z frame #6: PyObject_VectorcallMethod + 0x85 (0x56060cdbf215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5835217Z frame #7: <unknown function> + 0x208ae7 (0x56060ce72ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5835433Z frame #8: <unknown function> + 0x208472 (0x56060ce72472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5835652Z frame #9: <unknown function> + 0x1da6ee (0x56060ce446ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5835878Z frame #10: <unknown function> + 0x1483bc (0x56060cdb23bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5836115Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5836340Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5836573Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x56060cd963bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5836791Z frame #14: <unknown function> + 0x147734 (0x56060cdb1734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5837114Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x56060cd98c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5837398Z frame #16: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5837641Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5837923Z frame #18: _PyFunction_Vectorcall + 0x6c (0x56060cda60ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5838156Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x56060cd967b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5838443Z frame #20: <unknown function> + 0x147734 (0x56060cdb1734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5838660Z frame #21: <unknown function> + 0x21ca82 (0x56060ce86a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5838879Z frame #22: <unknown function> + 0x21c9e4 (0x56060ce869e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5839093Z frame #23: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5846359Z frame #24: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5846375Z 
2025-07-17T11:07:26.5846380Z 
2025-07-17T11:07:26.5846494Z SIGABRT(6), PID: 1811296, Thread 1816180: 
2025-07-17T11:07:26.5847012Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5847244Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5847456Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5847672Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5848076Z frame #4: <unknown function> + 0x5a02023 (0x7f1dbb609023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5848664Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1dbb60d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5849262Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1dbb6041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5849654Z frame #7: <unknown function> + 0x79d655 (0x7f1dccaf4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5849896Z frame #8: <unknown function> + 0xdc253 (0x7f1d83b3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5850114Z frame #9: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5850331Z frame #10: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5850338Z 
2025-07-17T11:07:26.5850438Z SIGABRT(6), PID: 1811300, Thread 1816189: 
2025-07-17T11:07:26.5850923Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5851140Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5851346Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5851555Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5851931Z frame #4: <unknown function> + 0x5a02023 (0x7f2171f59023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5852496Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2171f5d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5853279Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2171f541d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5853730Z frame #7: <unknown function> + 0x79d655 (0x7f2183444655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5853975Z frame #8: <unknown function> + 0xdc253 (0x7f213a48f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5854259Z frame #9: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5854476Z frame #10: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5854482Z 
2025-07-17T11:07:26.5854581Z SIGABRT(6), PID: 1811294, Thread 1816152: 
2025-07-17T11:07:26.5855058Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5855271Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5855475Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5855688Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5856147Z frame #4: <unknown function> + 0x5a02023 (0x7f7b1b33c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5856726Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7b1b3401ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5857323Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7b1b3371d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5857709Z frame #7: <unknown function> + 0x79d655 (0x7f7b2c827655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5857943Z frame #8: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5858154Z frame #9: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5858371Z frame #10: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5858377Z 
2025-07-17T11:07:26.5858469Z SIGABRT(6), PID: 1811296, Thread 1816181: 
2025-07-17T11:07:26.5858946Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5859154Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5859361Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5859564Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5859938Z frame #4: <unknown function> + 0x5a02023 (0x7f1dbb609023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5860497Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1dbb60d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5861086Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1dbb6041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5861462Z frame #7: <unknown function> + 0x79d655 (0x7f1dccaf4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5861836Z frame #8: <unknown function> + 0xdc253 (0x7f1d83b3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5862123Z frame #9: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5862342Z frame #10: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5862346Z 
2025-07-17T11:07:26.5862508Z SIGABRT(6), PID: 1811294, Thread 1816153: 
2025-07-17T11:07:26.5862985Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5863273Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5863478Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5863680Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5864054Z frame #4: <unknown function> + 0x5a02023 (0x7f7b1b33c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5864611Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7b1b3401ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5865195Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7b1b3371d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5865570Z frame #7: <unknown function> + 0x79d655 (0x7f7b2c827655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5865799Z frame #8: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5866008Z frame #9: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5866223Z frame #10: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5866227Z 
2025-07-17T11:07:26.5866319Z SIGABRT(6), PID: 1811300, Thread 1816190: 
2025-07-17T11:07:26.5866791Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5867001Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5867204Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5867405Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5867775Z frame #4: <unknown function> + 0x5a02023 (0x7f2171f59023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5868344Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2171f5d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5868932Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2171f541d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5869318Z frame #7: <unknown function> + 0x79d655 (0x7f2183444655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5869545Z frame #8: <unknown function> + 0xdc253 (0x7f213a48f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5869758Z frame #9: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5869970Z frame #10: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5870051Z 
2025-07-17T11:07:26.5870149Z SIGABRT(6), PID: 1811296, Thread 1816182: 
2025-07-17T11:07:26.5870810Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5871021Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5871283Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5871547Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5871916Z frame #4: <unknown function> + 0x5a02023 (0x7f1dbb609023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5872477Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1dbb60d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5873066Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1dbb6041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5873451Z frame #7: <unknown function> + 0x79d655 (0x7f1dccaf4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5873680Z frame #8: <unknown function> + 0xdc253 (0x7f1d83b3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5873890Z frame #9: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5874103Z frame #10: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5874107Z 
2025-07-17T11:07:26.5874198Z SIGABRT(6), PID: 1811294, Thread 1816154: 
2025-07-17T11:07:26.5874677Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5874890Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5875094Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5875298Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5875672Z frame #4: <unknown function> + 0x5a02023 (0x7f7b1b33c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5876231Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7b1b3401ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5876820Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7b1b3371d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5877198Z frame #7: <unknown function> + 0x79d655 (0x7f7b2c827655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5877427Z frame #8: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5877645Z frame #9: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5877860Z frame #10: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5877865Z 
2025-07-17T11:07:26.5877954Z SIGABRT(6), PID: 1811300, Thread 1816191: 
2025-07-17T11:07:26.5878428Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f213c222b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5878634Z frame #1: <unknown function> + 0x42520 (0x7f222dad9520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5878914Z frame #2: <unknown function> + 0x91117 (0x7f222db28117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5879173Z frame #3: pthread_cond_wait + 0x211 (0x7f222db2aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5879546Z frame #4: <unknown function> + 0x5a02023 (0x7f2171f59023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5880160Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f2171f5d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5880801Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f2171f541d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5881176Z frame #7: <unknown function> + 0x79d655 (0x7f2183444655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5881405Z frame #8: <unknown function> + 0xdc253 (0x7f213a48f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5881616Z frame #9: <unknown function> + 0x94ac3 (0x7f222db2bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5881830Z frame #10: <unknown function> + 0x126850 (0x7f222dbbd850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5881835Z 
2025-07-17T11:07:26.5881925Z SIGABRT(6), PID: 1811296, Thread 1816185: 
2025-07-17T11:07:26.5882399Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1d858d2b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5882605Z frame #1: <unknown function> + 0x42520 (0x7f1e7718c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5882809Z frame #2: <unknown function> + 0x91117 (0x7f1e771db117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5883007Z frame #3: pthread_cond_wait + 0x211 (0x7f1e771dda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5883379Z frame #4: <unknown function> + 0x5a02023 (0x7f1dbb609023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5883933Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1dbb60d1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5884520Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1dbb6041d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5884899Z frame #7: <unknown function> + 0x79d655 (0x7f1dccaf4655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5885125Z frame #8: <unknown function> + 0xdc253 (0x7f1d83b3f253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5885336Z frame #9: <unknown function> + 0x94ac3 (0x7f1e771deac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5885549Z frame #10: <unknown function> + 0x126850 (0x7f1e77270850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5885553Z 
2025-07-17T11:07:26.5885644Z SIGABRT(6), PID: 1811294, Thread 1816155: 
2025-07-17T11:07:26.5886113Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5886322Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5886526Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5886725Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5887092Z frame #4: <unknown function> + 0x5a02023 (0x7f7b1b33c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5887792Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7b1b3401ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5888438Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7b1b3371d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5888818Z frame #7: <unknown function> + 0x79d655 (0x7f7b2c827655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5889131Z frame #8: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5889344Z frame #9: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5889560Z frame #10: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5889566Z 
2025-07-17T11:07:26.5889659Z SIGABRT(6), PID: 1811294, Thread 1816156: 
2025-07-17T11:07:26.5890127Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5890339Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5890544Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5890747Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5891114Z frame #4: <unknown function> + 0x5a02023 (0x7f7b1b33c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5891674Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7b1b3401ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5892263Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7b1b3371d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5892639Z frame #7: <unknown function> + 0x79d655 (0x7f7b2c827655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5892865Z frame #8: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5893078Z frame #9: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5893291Z frame #10: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5893298Z 
2025-07-17T11:07:26.5893387Z SIGABRT(6), PID: 1811294, Thread 1816157: 
2025-07-17T11:07:26.5893860Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5894073Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5894278Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5894474Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5894844Z frame #4: <unknown function> + 0x5a02023 (0x7f7b1b33c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5895399Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7b1b3401ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5896058Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7b1b3371d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5896598Z frame #7: <unknown function> + 0x79d655 (0x7f7b2c827655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5896832Z frame #8: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5897113Z frame #9: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5897332Z frame #10: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5897407Z 
2025-07-17T11:07:26.5897501Z SIGABRT(6), PID: 1811294, Thread 1816158: 
2025-07-17T11:07:26.5897967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5898174Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5898382Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5898583Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5898951Z frame #4: <unknown function> + 0x5a02023 (0x7f7b1b33c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5899507Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7b1b3401ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5900112Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7b1b3371d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5900504Z frame #7: <unknown function> + 0x79d655 (0x7f7b2c827655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5900740Z frame #8: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5900957Z frame #9: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5901178Z frame #10: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5901182Z 
2025-07-17T11:07:26.5901275Z SIGABRT(6), PID: 1811294, Thread 1816159: 
2025-07-17T11:07:26.5901757Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7ae5605b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5901972Z frame #1: <unknown function> + 0x42520 (0x7f7bd6ebd520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5902182Z frame #2: <unknown function> + 0x91117 (0x7f7bd6f0c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5902388Z frame #3: pthread_cond_wait + 0x211 (0x7f7bd6f0ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5902768Z frame #4: <unknown function> + 0x5a02023 (0x7f7b1b33c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5903333Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7b1b3401ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5903926Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7b1b3371d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.5904308Z frame #7: <unknown function> + 0x79d655 (0x7f7b2c827655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5904540Z frame #8: <unknown function> + 0xdc253 (0x7f7ae3872253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.5904831Z frame #9: <unknown function> + 0x94ac3 (0x7f7bd6f0fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5905109Z frame #10: <unknown function> + 0x126850 (0x7f7bd6fa1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5905113Z 
2025-07-17T11:07:26.5905200Z FAILED [25.5835s] [100%]
2025-07-17T11:07:26.5905204Z 
2025-07-17T11:07:26.5905314Z ==================================== RERUNS ====================================
2025-07-17T11:07:26.5905557Z __________________ TestMultiProc.test_ddp_optimizer_cudagraph __________________
2025-07-17T11:07:26.5905710Z Traceback (most recent call last):
2025-07-17T11:07:26.5906102Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in wrapper
2025-07-17T11:07:26.5906190Z     self._join_processes(fn)
2025-07-17T11:07:26.5906593Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 920, in _join_processes
2025-07-17T11:07:26.5906706Z     self._check_return_codes(fn, elapsed_time)
2025-07-17T11:07:26.5907119Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 996, in _check_return_codes
2025-07-17T11:07:26.5907199Z     self.assertEqual(
2025-07-17T11:07:26.5907560Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4176, in assertEqual
2025-07-17T11:07:26.5907712Z     raise error_metas.pop()[0].to_error(  # type: ignore[index]
2025-07-17T11:07:26.5907813Z AssertionError: Scalars are not equal!
2025-07-17T11:07:26.5907819Z 
2025-07-17T11:07:26.5907904Z Expected 0 but got -6.
2025-07-17T11:07:26.5907986Z Absolute difference: 6
2025-07-17T11:07:26.5908075Z Relative difference: inf
2025-07-17T11:07:26.5908193Z Expected exit code 0 but got -6 for pid: 1800818
2025-07-17T11:07:26.5908378Z __________________ TestMultiProc.test_ddp_optimizer_cudagraph __________________
2025-07-17T11:07:26.5908473Z Traceback (most recent call last):
2025-07-17T11:07:26.5908844Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in wrapper
2025-07-17T11:07:26.5908933Z     self._join_processes(fn)
2025-07-17T11:07:26.5909327Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 920, in _join_processes
2025-07-17T11:07:26.5909431Z     self._check_return_codes(fn, elapsed_time)
2025-07-17T11:07:26.5909839Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 996, in _check_return_codes
2025-07-17T11:07:26.5909919Z     self.assertEqual(
2025-07-17T11:07:26.5910278Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4176, in assertEqual
2025-07-17T11:07:26.5910424Z     raise error_metas.pop()[0].to_error(  # type: ignore[index]
2025-07-17T11:07:26.5910525Z AssertionError: Scalars are not equal!
2025-07-17T11:07:26.5910530Z 
2025-07-17T11:07:26.5910608Z Expected 0 but got -6.
2025-07-17T11:07:26.5910690Z Absolute difference: 6
2025-07-17T11:07:26.5910769Z Relative difference: inf
2025-07-17T11:07:26.5910889Z Expected exit code 0 but got -6 for pid: 1805788
2025-07-17T11:07:26.5910997Z =================================== FAILURES ===================================
2025-07-17T11:07:26.5911176Z __________________ TestMultiProc.test_ddp_optimizer_cudagraph __________________
2025-07-17T11:07:26.5911267Z Traceback (most recent call last):
2025-07-17T11:07:26.5911633Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in wrapper
2025-07-17T11:07:26.5911718Z     self._join_processes(fn)
2025-07-17T11:07:26.5912114Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 920, in _join_processes
2025-07-17T11:07:26.5912215Z     self._check_return_codes(fn, elapsed_time)
2025-07-17T11:07:26.5912700Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 996, in _check_return_codes
2025-07-17T11:07:26.5912777Z     self.assertEqual(
2025-07-17T11:07:26.5913199Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4176, in assertEqual
2025-07-17T11:07:26.5913344Z     raise error_metas.pop()[0].to_error(  # type: ignore[index]
2025-07-17T11:07:26.5913498Z AssertionError: Scalars are not equal!
2025-07-17T11:07:26.5913503Z 
2025-07-17T11:07:26.5913585Z Expected 0 but got -6.
2025-07-17T11:07:26.5913726Z Absolute difference: 6
2025-07-17T11:07:26.5913807Z Relative difference: inf
2025-07-17T11:07:26.5913921Z Expected exit code 0 but got -6 for pid: 1811294
2025-07-17T11:07:26.5914509Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-cf10226461b1eeac.xml -
2025-07-17T11:07:26.5914637Z =========================== short test summary info ============================
2025-07-17T11:07:26.5915052Z FAILED [25.5835s] distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph - AssertionError: Scalars are not equal!
2025-07-17T11:07:26.5915056Z 
2025-07-17T11:07:26.5915141Z Expected 0 but got -6.
2025-07-17T11:07:26.5915219Z Absolute difference: 6
2025-07-17T11:07:26.5915298Z Relative difference: inf
2025-07-17T11:07:26.5915413Z Expected exit code 0 but got -6 for pid: 1811294
2025-07-17T11:07:26.5915540Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-07-17T11:07:26.5915689Z ============= 1 failed, 22 deselected, 2 rerun in 76.91s (0:01:16) =============
2025-07-17T11:07:26.5916237Z /opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 24 leaked semaphore objects to clean up at shutdown
2025-07-17T11:07:26.5916382Z   warnings.warn('resource_tracker: There appear to be %d '
2025-07-17T11:07:26.5916458Z Got exit code 1
2025-07-17T11:07:26.5916541Z Retrying single test...
2025-07-17T11:07:26.5917560Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.5917648Z   import pkg_resources
2025-07-17T11:07:26.5918118Z Test results will be stored in test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-28cda1fd8ef32fac.xml
2025-07-17T11:07:26.5918240Z ============================= test session starts ==============================
2025-07-17T11:07:26.5918486Z platform linux -- Python 3.10.18, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python
2025-07-17T11:07:26.5918572Z cachedir: .pytest_cache
2025-07-17T11:07:26.5918929Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-07-17T11:07:26.5919025Z rootdir: /var/lib/jenkins/pytorch
2025-07-17T11:07:26.5919107Z configfile: pytest.ini
2025-07-17T11:07:26.5919474Z plugins: xdist-3.3.1, rerunfailures-14.0, xdoctest-1.1.0, flakefinder-1.1.0, cpp-2.3.0, subtests-0.13.1, hypothesis-5.35.1, typeguard-4.3.0
2025-07-17T11:07:26.5919629Z collecting ... collected 60 items / 22 deselected / 38 selected
2025-07-17T11:07:26.5920062Z stepcurrent: skipping 8 already run items. Running only test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph
2025-07-17T11:07:26.5920150Z Running 1 items in this shard
2025-07-17T11:07:26.5920154Z 
2025-07-17T11:07:26.5920803Z distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph I0717 11:02:34.753000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1817866
2025-07-17T11:07:26.5921236Z I0717 11:02:34.754000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1817867
2025-07-17T11:07:26.5921643Z I0717 11:02:34.755000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1817868
2025-07-17T11:07:26.5921990Z I0717 11:02:34.756000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1817869
2025-07-17T11:07:26.5922417Z I0717 11:02:34.757000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1817870
2025-07-17T11:07:26.5922825Z I0717 11:02:34.758000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1817871
2025-07-17T11:07:26.5923167Z I0717 11:02:34.759000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1817872
2025-07-17T11:07:26.5923508Z I0717 11:02:34.760000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1817873
2025-07-17T11:07:26.5924498Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.5924579Z   import pkg_resources
2025-07-17T11:07:26.5925546Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.5925627Z   import pkg_resources
2025-07-17T11:07:26.5926590Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.5926673Z   import pkg_resources
2025-07-17T11:07:26.5927637Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.5927723Z   import pkg_resources
2025-07-17T11:07:26.5928693Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.5928774Z   import pkg_resources
2025-07-17T11:07:26.5929737Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.5929815Z   import pkg_resources
2025-07-17T11:07:26.5930781Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.5930858Z   import pkg_resources
2025-07-17T11:07:26.5931828Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.5932038Z   import pkg_resources
2025-07-17T11:07:26.5932525Z [rank7]:W0717 11:02:56.680000 1817873 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.5933055Z [rank0]:W0717 11:02:56.681000 1817866 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.5933582Z [rank4]:W0717 11:02:56.683000 1817870 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.5934043Z [rank6]:W0717 11:02:56.683000 1817872 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.5934509Z [rank5]:W0717 11:02:56.684000 1817871 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.5934968Z [rank3]:W0717 11:02:56.684000 1817869 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.5935430Z [rank2]:W0717 11:02:56.685000 1817868 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.5935891Z [rank1]:W0717 11:02:56.686000 1817867 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.5936246Z Memory access fault by GPU node-4 (Agent handle: 0x55937ad53cd0) on address 0x7f60d2a03000. Reason: Unknown.
2025-07-17T11:07:26.5936518Z Memory access fault by GPU node-8 (Agent handle: 0x5626b2622040) on address 0x7fd2da803000. Reason: Unknown.
2025-07-17T11:07:26.5936796Z Memory access fault by GPU node-6 (Agent handle: 0x563c78a8c9b0) on address 0x7f61ebe1d000. Reason: Unknown.
2025-07-17T11:07:26.5937061Z Memory access fault by GPU node-9 (Agent handle: 0x55b7e08476f0) on address 0x7f3fa6003000. Reason: Unknown.
2025-07-17T11:07:26.5937328Z Memory access fault by GPU node-5 (Agent handle: 0x55f863bc9bd0) on address 0x7ff066a23000. Reason: Unknown.
2025-07-17T11:07:26.5937595Z Memory access fault by GPU node-3 (Agent handle: 0x56497f376210) on address 0x7f0a2a61f000. Reason: Unknown.
2025-07-17T11:07:26.5937865Z Memory access fault by GPU node-7 (Agent handle: 0x556e65ca2c20) on address 0x7f9a0ea1f000. Reason: Unknown.
2025-07-17T11:07:26.5938125Z Memory access fault by GPU node-2 (Agent handle: 0x56225e75e9e0) on address 0x7fe457406000. Reason: Unknown.
2025-07-17T11:07:26.5938226Z SIGABRT(6), PID: 1817870, Thread 1817870: 
2025-07-17T11:07:26.5938715Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5938942Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5939178Z frame #2: <unknown function> + 0x77435 (0x7f6a304c6435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5939421Z frame #3: <unknown function> + 0x76c8e (0x7f6a304c5c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5939649Z frame #4: <unknown function> + 0x6ad69 (0x7f6a304b9d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5939867Z frame #5: <unknown function> + 0x3f54db (0x7f6ac111e4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5940080Z frame #6: <unknown function> + 0x3e11d8 (0x7f6ac110a1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5940294Z frame #7: <unknown function> + 0xb2f86 (0x7f6ac0ddbf86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5940602Z frame #8: <unknown function> + 0x8e501 (0x7f6ac0db7501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5941072Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f6aebbbe7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.5941536Z frame #10: <unknown function> + 0xd307d9 (0x7f6b0a1117d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5941769Z frame #11: <unknown function> + 0x139902 (0x563c6e1c2902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5942117Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x563c6e1b9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5942351Z frame #13: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5942595Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x563c6e1b9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5942823Z frame #15: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5943065Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5943290Z frame #17: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5943528Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5943755Z frame #19: <unknown function> + 0x1474b2 (0x563c6e1d04b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5943959Z frame #20: PyObject_Call + 0xbc (0x563c6e1d0e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5944182Z frame #21: <unknown function> + 0x1ff69d (0x563c6e28869d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5944408Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x563c6e1be023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5944655Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x563c6e1b9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5944878Z frame #24: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5945118Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x563c6e1b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5945342Z frame #26: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5945580Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x563c6e1b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5945804Z frame #28: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5946048Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x563c6e1b9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5946306Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x563c6e1bd4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5946530Z frame #31: _PyObject_Call_Prepend + 0x69 (0x563c6e1ceab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5946749Z frame #32: <unknown function> + 0x205949 (0x563c6e28e949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5946980Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x563c6e1be023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5947220Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x563c6e1b9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5947446Z frame #35: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5947679Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x563c6e1b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5947906Z frame #37: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5948140Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x563c6e1b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5948364Z frame #39: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5948675Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x563c6e1b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5948969Z frame #41: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5949212Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x563c6e1b63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5949500Z frame #43: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5949894Z frame #44: <unknown function> + 0x7ab3f7 (0x7f6b09b8c3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5950179Z frame #45: <unknown function> + 0x13bc88 (0x563c6e1c4c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5950384Z frame #46: PyObject_Call + 0x20f (0x563c6e1d0f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5950630Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x563c6e1ba98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5950851Z frame #48: <unknown function> + 0x147608 (0x563c6e1d0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5951092Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x563c6e1b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5951313Z frame #50: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5951553Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x563c6e1b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5951778Z frame #52: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5952016Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x563c6e1b63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5952236Z frame #54: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5952477Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x563c6e1b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5952696Z frame #56: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5952937Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x563c6e1b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5953156Z frame #58: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5953397Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x563c6e1b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5953614Z frame #60: <unknown function> + 0x147608 (0x563c6e1d0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5953851Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x563c6e1b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5954063Z frame #62: <unknown function> + 0x147608 (0x563c6e1d0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5954072Z 
2025-07-17T11:07:26.5954168Z SIGABRT(6), PID: 1817870, Thread 1818099: 
2025-07-17T11:07:26.5954652Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5954868Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5955078Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5955279Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5955798Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5956008Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5956224Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5956300Z 
2025-07-17T11:07:26.5956396Z SIGABRT(6), PID: 1817870, Thread 1818103: 
2025-07-17T11:07:26.5956958Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5957173Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5957441Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5957644Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5958232Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5958447Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5958666Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5958670Z 
2025-07-17T11:07:26.5958760Z SIGABRT(6), PID: 1817870, Thread 1818109: 
2025-07-17T11:07:26.5959244Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5959452Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5959660Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5959862Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5960368Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5960575Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5960789Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5960793Z 
2025-07-17T11:07:26.5960883Z SIGABRT(6), PID: 1817870, Thread 1818113: 
2025-07-17T11:07:26.5961360Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5961564Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5961772Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5961970Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5962471Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5962681Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5962893Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5962898Z 
2025-07-17T11:07:26.5962991Z SIGABRT(6), PID: 1817870, Thread 1818116: 
2025-07-17T11:07:26.5963466Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5963670Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5963876Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5964071Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5964566Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5964856Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5965133Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5965137Z 
2025-07-17T11:07:26.5965232Z SIGABRT(6), PID: 1817870, Thread 1818120: 
2025-07-17T11:07:26.5965758Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5966025Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5966230Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5966430Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5966926Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5967137Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5967345Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5967349Z 
2025-07-17T11:07:26.5967443Z SIGABRT(6), PID: 1817870, Thread 1818124: 
2025-07-17T11:07:26.5967912Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5968121Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5968320Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5968519Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5969017Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5969227Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5969439Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5969444Z 
2025-07-17T11:07:26.5969535Z SIGABRT(6), PID: 1817870, Thread 1818128: 
2025-07-17T11:07:26.5970003Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5970209Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5970411Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5970613Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5971104Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5971312Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5971520Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5971524Z 
2025-07-17T11:07:26.5971616Z SIGABRT(6), PID: 1817870, Thread 1818132: 
2025-07-17T11:07:26.5972090Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5972299Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5972579Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5972782Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5973341Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5973617Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5973829Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5973893Z 
2025-07-17T11:07:26.5973989Z SIGABRT(6), PID: 1817870, Thread 1818135: 
2025-07-17T11:07:26.5974463Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5974671Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5974876Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5975074Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5975572Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5975784Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5976052Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5976060Z 
2025-07-17T11:07:26.5976150Z SIGABRT(6), PID: 1817868, Thread 1817868: 
2025-07-17T11:07:26.5976624Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5976829Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5977064Z frame #2: <unknown function> + 0x77445 (0x7f690e8db445 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5977293Z frame #3: <unknown function> + 0x76c8e (0x7f690e8dac8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5977519Z frame #4: <unknown function> + 0x6ad69 (0x7f690e8ced69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.5977742Z frame #5: <unknown function> + 0x3f54db (0x7f699f5334db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5977958Z frame #6: <unknown function> + 0x3e11d8 (0x7f699f51f1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5978169Z frame #7: <unknown function> + 0xb2f86 (0x7f699f1f0f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5978379Z frame #8: <unknown function> + 0x8e501 (0x7f699f1cc501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.5978766Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f69c9fd37ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.5979158Z frame #10: <unknown function> + 0xd307d9 (0x7f69e851d7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5979384Z frame #11: <unknown function> + 0x139902 (0x559370a38902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5979635Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x559370a2fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5979866Z frame #13: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5980110Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x559370a2fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5980333Z frame #15: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5980577Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5980897Z frame #17: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5981207Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5981431Z frame #19: <unknown function> + 0x1474b2 (0x559370a464b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5981706Z frame #20: PyObject_Call + 0xbc (0x559370a46e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5981931Z frame #21: <unknown function> + 0x1ff69d (0x559370afe69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5982227Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x559370a34023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5982470Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x559370a2fcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5982696Z frame #24: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5982932Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x559370a2b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5983154Z frame #26: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5983388Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x559370a2b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5983611Z frame #28: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5983852Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x559370a2fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5984111Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x559370a334b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5984327Z frame #31: _PyObject_Call_Prepend + 0x69 (0x559370a44ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5984547Z frame #32: <unknown function> + 0x205949 (0x559370b04949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5984769Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x559370a34023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5985011Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x559370a2fcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5985240Z frame #35: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5985480Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x559370a2b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5985701Z frame #37: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5985941Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x559370a2b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5986163Z frame #39: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5986397Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x559370a2b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5986621Z frame #41: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5986860Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x559370a2c3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5987087Z frame #43: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5987480Z frame #44: <unknown function> + 0x7ab3f7 (0x7f69e7f983f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.5987700Z frame #45: <unknown function> + 0x13bc88 (0x559370a3ac88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5987906Z frame #46: PyObject_Call + 0x20f (0x559370a46f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5988145Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x559370a3098d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5988362Z frame #48: <unknown function> + 0x147608 (0x559370a46608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5988697Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x559370a2dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5988987Z frame #50: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5989227Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x559370a2b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5989454Z frame #52: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5989754Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x559370a2c3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5990038Z frame #54: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5990276Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x559370a2b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5990502Z frame #56: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5990743Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x559370a2dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5990967Z frame #58: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5991206Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x559370a2dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5991425Z frame #60: <unknown function> + 0x147608 (0x559370a46608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5991665Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x559370a2dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5991881Z frame #62: <unknown function> + 0x147608 (0x559370a46608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.5991977Z SIGABRT(6), PID: 1817870, Thread 1818138: 
2025-07-17T11:07:26.5992461Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5992676Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5992887Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5993091Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5993601Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5993810Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5994023Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5994027Z 
2025-07-17T11:07:26.5994031Z 
2025-07-17T11:07:26.5994123Z SIGABRT(6), PID: 1817870, Thread 1818139: 
2025-07-17T11:07:26.5994601Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5994808Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5995014Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5995211Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5995707Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5995915Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5996127Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5996218Z SIGABRT(6), PID: 1817868, Thread 1817895: 
2025-07-17T11:07:26.5996763Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5997032Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5997243Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5997440Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5997995Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.5998262Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5998474Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5998478Z 
2025-07-17T11:07:26.5998483Z 
2025-07-17T11:07:26.5998572Z SIGABRT(6), PID: 1817870, Thread 1818141: 
2025-07-17T11:07:26.5999042Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.5999247Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5999453Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.5999655Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6000164Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6000373Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6000584Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6000590Z 
2025-07-17T11:07:26.6000679Z SIGABRT(6), PID: 1817868, Thread 1817898: 
2025-07-17T11:07:26.6001151Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6001361Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6001567Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6001765Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6002263Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6002473Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6002683Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6002687Z 
2025-07-17T11:07:26.6002777Z SIGABRT(6), PID: 1817870, Thread 1818143: 
2025-07-17T11:07:26.6003248Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6003457Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6003661Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6003863Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6004355Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6004563Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6004964Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6005116Z SIGABRT(6), PID: 1817868, Thread 1817901: 
2025-07-17T11:07:26.6005587Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6005856Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6006061Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6006320Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6006819Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6007032Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6007242Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6007248Z 
2025-07-17T11:07:26.6007251Z 
2025-07-17T11:07:26.6007341Z SIGABRT(6), PID: 1817870, Thread 1818145: 
2025-07-17T11:07:26.6007812Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6008019Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6008222Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6008421Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6008915Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6009125Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6009336Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6009339Z 
2025-07-17T11:07:26.6009431Z SIGABRT(6), PID: 1817868, Thread 1817903: 
2025-07-17T11:07:26.6009900Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6010108Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6010311Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6010513Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6011005Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6011216Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6011422Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6011428Z 
2025-07-17T11:07:26.6011518Z SIGABRT(6), PID: 1817870, Thread 1818147: 
2025-07-17T11:07:26.6011987Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6012197Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6012402Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6012598Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6013254Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6013464Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6013676Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6013741Z 
2025-07-17T11:07:26.6013833Z SIGABRT(6), PID: 1817868, Thread 1817906: 
2025-07-17T11:07:26.6014388Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6014591Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6014794Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6014993Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6015490Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6015696Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6015972Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6015976Z 
2025-07-17T11:07:26.6016070Z SIGABRT(6), PID: 1817870, Thread 1818149: 
2025-07-17T11:07:26.6016537Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6016743Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6016948Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6017144Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6017644Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6017852Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6018066Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6018072Z 
2025-07-17T11:07:26.6018161Z SIGABRT(6), PID: 1817868, Thread 1817908: 
2025-07-17T11:07:26.6018635Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6018838Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6019043Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6019241Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6019733Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6019942Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6020152Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6020157Z 
2025-07-17T11:07:26.6020244Z SIGABRT(6), PID: 1817870, Thread 1818151: 
2025-07-17T11:07:26.6020713Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6021012Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6021218Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6021489Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6022082Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6022293Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6022583Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6022673Z SIGABRT(6), PID: 1817868, Thread 1817910: 
2025-07-17T11:07:26.6023140Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6023350Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6023553Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6023751Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6024247Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6024455Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6024664Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6024668Z 
2025-07-17T11:07:26.6024671Z 
2025-07-17T11:07:26.6024762Z SIGABRT(6), PID: 1817868, Thread 1817911: 
2025-07-17T11:07:26.6025230Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6025439Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6025640Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6025840Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6026336Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6026545Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6026756Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6026846Z SIGABRT(6), PID: 1817870, Thread 1818153: 
2025-07-17T11:07:26.6027312Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6027520Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6027724Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6027929Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6028418Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6028629Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6028837Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6028912Z 
2025-07-17T11:07:26.6028915Z 
2025-07-17T11:07:26.6029014Z SIGABRT(6), PID: 1817870, Thread 1818155: 
2025-07-17T11:07:26.6029542Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6029754Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6030020Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6030222Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6030777Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6030985Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6031195Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6031200Z 
2025-07-17T11:07:26.6031293Z SIGABRT(6), PID: 1817868, Thread 1817914: 
2025-07-17T11:07:26.6031762Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6031974Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6032177Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6032378Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6032872Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6033086Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6033294Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6033300Z 
2025-07-17T11:07:26.6033390Z SIGABRT(6), PID: 1817870, Thread 1818157: 
2025-07-17T11:07:26.6033857Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6034063Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6034270Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6034467Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6034958Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6035169Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6035379Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6035470Z SIGABRT(6), PID: 1817868, Thread 1817916: 
2025-07-17T11:07:26.6035940Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6036142Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6036346Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6036541Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6037033Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6037319Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6037593Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6037598Z 
2025-07-17T11:07:26.6037601Z 
2025-07-17T11:07:26.6037694Z SIGABRT(6), PID: 1817870, Thread 1818159: 
2025-07-17T11:07:26.6038225Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6038488Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6038696Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6038895Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6039389Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6039600Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6039810Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6039814Z 
2025-07-17T11:07:26.6039902Z SIGABRT(6), PID: 1817868, Thread 1817919: 
2025-07-17T11:07:26.6040374Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6040581Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6040790Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6040986Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6041488Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6041696Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6041905Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6041909Z 
2025-07-17T11:07:26.6041999Z SIGABRT(6), PID: 1817870, Thread 1818161: 
2025-07-17T11:07:26.6042469Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6042677Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6042886Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6043084Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6043580Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6043788Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6044000Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6044004Z 
2025-07-17T11:07:26.6044097Z SIGABRT(6), PID: 1817868, Thread 1817922: 
2025-07-17T11:07:26.6044567Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6044777Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6044978Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6045249Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6045798Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6046013Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6046299Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6046362Z 
2025-07-17T11:07:26.6046457Z SIGABRT(6), PID: 1817870, Thread 1818163: 
2025-07-17T11:07:26.6046930Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6047137Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6047340Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6047539Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6048036Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6048247Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6048455Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6048459Z 
2025-07-17T11:07:26.6048549Z SIGABRT(6), PID: 1817868, Thread 1817925: 
2025-07-17T11:07:26.6049015Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6049224Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6049424Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6049624Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6050119Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6050327Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6050537Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6050542Z 
2025-07-17T11:07:26.6050633Z SIGABRT(6), PID: 1817870, Thread 1818165: 
2025-07-17T11:07:26.6051098Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6051309Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6051513Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6051710Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6052203Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6052411Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6052618Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6052622Z 
2025-07-17T11:07:26.6052715Z SIGABRT(6), PID: 1817868, Thread 1817928: 
2025-07-17T11:07:26.6053180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6053519Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6053724Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6053926Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6054479Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6054756Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6054965Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6054969Z 
2025-07-17T11:07:26.6055058Z SIGABRT(6), PID: 1817870, Thread 1818168: 
2025-07-17T11:07:26.6055531Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6055738Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6055994Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6056195Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6056698Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6056905Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6057114Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6057119Z 
2025-07-17T11:07:26.6057207Z SIGABRT(6), PID: 1817868, Thread 1817932: 
2025-07-17T11:07:26.6057680Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6057885Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6058094Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6058290Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6058786Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6058994Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6059202Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6059208Z 
2025-07-17T11:07:26.6059297Z SIGABRT(6), PID: 1817870, Thread 1818171: 
2025-07-17T11:07:26.6059770Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6059973Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6060176Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6060377Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6060877Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6061088Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6061394Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6061398Z 
2025-07-17T11:07:26.6061486Z SIGABRT(6), PID: 1817868, Thread 1817935: 
2025-07-17T11:07:26.6062030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6062307Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6062514Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6062782Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6063278Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6063488Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6063700Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6063706Z 
2025-07-17T11:07:26.6063795Z SIGABRT(6), PID: 1817870, Thread 1818174: 
2025-07-17T11:07:26.6064266Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6064474Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6064677Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6064872Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6065369Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6065581Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6065791Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6065795Z 
2025-07-17T11:07:26.6065885Z SIGABRT(6), PID: 1817868, Thread 1817939: 
2025-07-17T11:07:26.6066351Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6066559Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6066760Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6066957Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6067450Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6067661Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6067869Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6067873Z 
2025-07-17T11:07:26.6067966Z SIGABRT(6), PID: 1817870, Thread 1818176: 
2025-07-17T11:07:26.6068432Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6068640Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6068842Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6069043Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6069609Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6069879Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6070090Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6070094Z 
2025-07-17T11:07:26.6070244Z SIGABRT(6), PID: 1817870, Thread 1818179: 
2025-07-17T11:07:26.6070729Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6071007Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6071213Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6071417Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6071920Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6072134Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6072344Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6072348Z 
2025-07-17T11:07:26.6072440Z SIGABRT(6), PID: 1817868, Thread 1817943: 
2025-07-17T11:07:26.6072909Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6073118Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6073318Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6073521Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6074018Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6074231Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6074441Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6074535Z SIGABRT(6), PID: 1817870, Thread 1818182: 
2025-07-17T11:07:26.6075004Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6075212Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6075416Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6075613Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6076110Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6076320Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6076529Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6076537Z 
2025-07-17T11:07:26.6076540Z 
2025-07-17T11:07:26.6076629Z SIGABRT(6), PID: 1817868, Thread 1817947: 
2025-07-17T11:07:26.6077098Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6077379Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6077589Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6077867Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6078435Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6078647Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6078919Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6078923Z 
2025-07-17T11:07:26.6079012Z SIGABRT(6), PID: 1817870, Thread 1818185: 
2025-07-17T11:07:26.6079479Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6079686Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6079892Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6080087Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6080585Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6080791Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6081004Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6081008Z 
2025-07-17T11:07:26.6081096Z SIGABRT(6), PID: 1817868, Thread 1817951: 
2025-07-17T11:07:26.6081566Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6081770Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6081976Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6082171Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6082666Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6082873Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6083082Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6083086Z 
2025-07-17T11:07:26.6083174Z SIGABRT(6), PID: 1817870, Thread 1818188: 
2025-07-17T11:07:26.6083649Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6083855Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6084059Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6084259Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6084757Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6084965Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6085177Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6085252Z 
2025-07-17T11:07:26.6085345Z SIGABRT(6), PID: 1817868, Thread 1817954: 
2025-07-17T11:07:26.6085871Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6086083Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6086285Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6086541Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6087090Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6087298Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6087505Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6087511Z 
2025-07-17T11:07:26.6087604Z SIGABRT(6), PID: 1817870, Thread 1818190: 
2025-07-17T11:07:26.6088072Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6088276Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6088478Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6088677Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6089169Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6089378Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6089586Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6089591Z 
2025-07-17T11:07:26.6089682Z SIGABRT(6), PID: 1817868, Thread 1817958: 
2025-07-17T11:07:26.6090146Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6090352Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6090553Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6090753Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6091243Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6091455Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6091661Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6091665Z 
2025-07-17T11:07:26.6091758Z SIGABRT(6), PID: 1817870, Thread 1818193: 
2025-07-17T11:07:26.6092222Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6092431Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6092636Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6092835Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6093327Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6093611Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6093884Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6093890Z 
2025-07-17T11:07:26.6093987Z SIGABRT(6), PID: 1817868, Thread 1817962: 
2025-07-17T11:07:26.6094517Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6094729Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6094999Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6095199Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6095690Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6095966Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6096180Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6096186Z 
2025-07-17T11:07:26.6096275Z SIGABRT(6), PID: 1817870, Thread 1818196: 
2025-07-17T11:07:26.6096749Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6096954Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6097158Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6097353Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6097853Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6098061Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6098270Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6098273Z 
2025-07-17T11:07:26.6098363Z SIGABRT(6), PID: 1817868, Thread 1817966: 
2025-07-17T11:07:26.6098836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6099042Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6099244Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6099441Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6099939Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6100146Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6100355Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6100361Z 
2025-07-17T11:07:26.6100449Z SIGABRT(6), PID: 1817870, Thread 1818198: 
2025-07-17T11:07:26.6100922Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6101126Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6101333Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6101626Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6102220Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6102431Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6102728Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6102808Z 
2025-07-17T11:07:26.6102903Z SIGABRT(6), PID: 1817872, Thread 1817872: 
2025-07-17T11:07:26.6103387Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6103599Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6103840Z frame #2: <unknown function> + 0x7743a (0x7fdb11dee43a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6104072Z frame #3: <unknown function> + 0x76c8e (0x7fdb11dedc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6104302Z frame #4: <unknown function> + 0x6ad69 (0x7fdb11de1d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6104519Z frame #5: <unknown function> + 0x3f54db (0x7fdba2a464db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6104735Z frame #6: <unknown function> + 0x3e11d8 (0x7fdba2a321d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6104945Z frame #7: <unknown function> + 0xb2f86 (0x7fdba2703f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6105155Z frame #8: <unknown function> + 0x8e501 (0x7fdba26df501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6105538Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fdbcd4e67ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.6105926Z frame #10: <unknown function> + 0xd307d9 (0x7fdbeba357d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6106157Z frame #11: <unknown function> + 0x139902 (0x5626a8d92902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6106410Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5626a8d89a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6106640Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6106883Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5626a8d89a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6107107Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6107347Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6107569Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6107812Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6108036Z frame #19: <unknown function> + 0x1474b2 (0x5626a8da04b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6108242Z frame #20: PyObject_Call + 0xbc (0x5626a8da0e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6108464Z frame #21: <unknown function> + 0x1ff69d (0x5626a8e5869d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6108692Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5626a8d8e023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6108935Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5626a8d89cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6109158Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6109394Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5626a8d853bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6109710Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6109944Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5626a8d853bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6110239Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6110481Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5626a8d89a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6110796Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5626a8d8d4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6111077Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5626a8d9eab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6111297Z frame #32: <unknown function> + 0x205949 (0x5626a8e5e949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6111518Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5626a8d8e023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6111760Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5626a8d89cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6111985Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6112225Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5626a8d853bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6112448Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6112685Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5626a8d853bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6112908Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6113145Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5626a8d853bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6113367Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6113610Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5626a8d863f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6113836Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6114228Z frame #44: <unknown function> + 0x7ab3f7 (0x7fdbeb4b03f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6114450Z frame #45: <unknown function> + 0x13bc88 (0x5626a8d94c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6114657Z frame #46: PyObject_Call + 0x20f (0x5626a8da0f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6114900Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5626a8d8a98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6115119Z frame #48: <unknown function> + 0x147608 (0x5626a8da0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6115358Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5626a8d87c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6115582Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6115817Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5626a8d853bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6116044Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6116282Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5626a8d863f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6116509Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6116743Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5626a8d853bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6116968Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6117206Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5626a8d87c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6117506Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6117745Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5626a8d87c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6118022Z frame #60: <unknown function> + 0x147608 (0x5626a8da0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6118264Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5626a8d87c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6118547Z frame #62: <unknown function> + 0x147608 (0x5626a8da0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6118607Z 
2025-07-17T11:07:26.6118703Z SIGABRT(6), PID: 1817870, Thread 1818202: 
2025-07-17T11:07:26.6119189Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6119401Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6119613Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6119814Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6120321Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6120530Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6120744Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6120750Z 
2025-07-17T11:07:26.6120841Z SIGABRT(6), PID: 1817868, Thread 1817971: 
2025-07-17T11:07:26.6121310Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6121516Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6121722Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6121920Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6122425Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6122633Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6122842Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6122846Z 
2025-07-17T11:07:26.6122938Z SIGABRT(6), PID: 1817870, Thread 1818205: 
2025-07-17T11:07:26.6123404Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6123611Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6123816Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6124014Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6124508Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6124719Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6124926Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6125019Z SIGABRT(6), PID: 1817872, Thread 1818166: 
2025-07-17T11:07:26.6125485Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6125773Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6126039Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6126249Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6126812Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6127201Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6127411Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6127415Z 
2025-07-17T11:07:26.6127419Z 
2025-07-17T11:07:26.6127511Z SIGABRT(6), PID: 1817868, Thread 1817976: 
2025-07-17T11:07:26.6127978Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6128186Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6128388Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6128586Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6129081Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6129294Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6129501Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6129507Z 
2025-07-17T11:07:26.6129599Z SIGABRT(6), PID: 1817872, Thread 1818169: 
2025-07-17T11:07:26.6130069Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6130277Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6130484Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6130686Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6131180Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6131391Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6131597Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6131695Z SIGABRT(6), PID: 1817868, Thread 1817981: 
2025-07-17T11:07:26.6132164Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6132371Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6132575Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6132772Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6133266Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6133473Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6133759Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6133766Z 
2025-07-17T11:07:26.6133769Z 
2025-07-17T11:07:26.6133859Z SIGABRT(6), PID: 1817870, Thread 1818209: 
2025-07-17T11:07:26.6134389Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6134657Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6134868Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6135149Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6135645Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6135856Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6136121Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6136126Z 
2025-07-17T11:07:26.6136220Z SIGABRT(6), PID: 1817872, Thread 1818172: 
2025-07-17T11:07:26.6136707Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6136921Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6137132Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6137334Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6137834Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6138045Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6138260Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6138264Z 
2025-07-17T11:07:26.6138354Z SIGABRT(6), PID: 1817868, Thread 1817984: 
2025-07-17T11:07:26.6138827Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6139031Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6139235Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6139432Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6139928Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6140137Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6140348Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6140437Z SIGABRT(6), PID: 1817870, Thread 1818212: 
2025-07-17T11:07:26.6140914Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6141119Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6141326Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6141524Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6142156Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6142449Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6142676Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6142768Z SIGABRT(6), PID: 1817872, Thread 1818175: 
2025-07-17T11:07:26.6143327Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6143614Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6143826Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6144031Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6144539Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6144752Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6144967Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6144972Z 
2025-07-17T11:07:26.6144977Z 
2025-07-17T11:07:26.6144980Z 
2025-07-17T11:07:26.6145073Z SIGABRT(6), PID: 1817870, Thread 1818216: 
2025-07-17T11:07:26.6145550Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6145765Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6145967Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6146170Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6146666Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6146877Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6147085Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6147091Z 
2025-07-17T11:07:26.6147186Z SIGABRT(6), PID: 1817868, Thread 1817989: 
2025-07-17T11:07:26.6147654Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6147860Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6148063Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6148260Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6148754Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6148966Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6149174Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6149180Z 
2025-07-17T11:07:26.6149271Z SIGABRT(6), PID: 1817872, Thread 1818178: 
2025-07-17T11:07:26.6149739Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6149947Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6150234Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6150496Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6150995Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6151268Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6151539Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6151543Z 
2025-07-17T11:07:26.6151641Z SIGABRT(6), PID: 1817868, Thread 1817993: 
2025-07-17T11:07:26.6152112Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6152322Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6152527Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6152725Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6153219Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6153428Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6153635Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6153642Z 
2025-07-17T11:07:26.6153731Z SIGABRT(6), PID: 1817870, Thread 1818220: 
2025-07-17T11:07:26.6154199Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6154407Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6154614Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6154808Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6155309Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6155516Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6155728Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6155732Z 
2025-07-17T11:07:26.6155821Z SIGABRT(6), PID: 1817872, Thread 1818181: 
2025-07-17T11:07:26.6156299Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6156507Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6156716Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6156914Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6157410Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6157619Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6157830Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6157920Z SIGABRT(6), PID: 1817868, Thread 1817996: 
2025-07-17T11:07:26.6158465Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6158731Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6158942Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6159203Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6159702Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6159972Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6160183Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6160189Z 
2025-07-17T11:07:26.6160193Z 
2025-07-17T11:07:26.6160281Z SIGABRT(6), PID: 1817870, Thread 1818224: 
2025-07-17T11:07:26.6160752Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6160957Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6161160Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6161354Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6161855Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6162064Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6162277Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6162281Z 
2025-07-17T11:07:26.6162370Z SIGABRT(6), PID: 1817868, Thread 1818001: 
2025-07-17T11:07:26.6162844Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6163052Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6163256Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6163451Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6163946Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6164151Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6164363Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6164456Z SIGABRT(6), PID: 1817872, Thread 1818183: 
2025-07-17T11:07:26.6164925Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6165133Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6165340Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6165539Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6166035Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6166323Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6166531Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6166594Z 
2025-07-17T11:07:26.6166598Z 
2025-07-17T11:07:26.6166693Z SIGABRT(6), PID: 1817870, Thread 1818228: 
2025-07-17T11:07:26.6167242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6167454Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6167719Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6167920Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6168413Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6168634Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6168843Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6168847Z 
2025-07-17T11:07:26.6168939Z SIGABRT(6), PID: 1817868, Thread 1818005: 
2025-07-17T11:07:26.6169406Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6169619Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6169823Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6170020Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6170521Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6170731Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6170939Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6170943Z 
2025-07-17T11:07:26.6171035Z SIGABRT(6), PID: 1817870, Thread 1818232: 
2025-07-17T11:07:26.6171505Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6171715Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6171916Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6172112Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6172609Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6172818Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6173024Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6173117Z SIGABRT(6), PID: 1817872, Thread 1818186: 
2025-07-17T11:07:26.6173587Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6173797Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6174000Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6174274Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6174828Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6175043Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6175309Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6175315Z 
2025-07-17T11:07:26.6175322Z 
2025-07-17T11:07:26.6175472Z SIGABRT(6), PID: 1817870, Thread 1818236: 
2025-07-17T11:07:26.6176001Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6176208Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6176414Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6176610Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6177107Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6177315Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6177525Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6177618Z SIGABRT(6), PID: 1817872, Thread 1818189: 
2025-07-17T11:07:26.6178089Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6178293Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6178503Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6178699Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6179193Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6179402Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6179613Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6179618Z 
2025-07-17T11:07:26.6179622Z 
2025-07-17T11:07:26.6179710Z SIGABRT(6), PID: 1817868, Thread 1818009: 
2025-07-17T11:07:26.6180176Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6180382Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6180588Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6180785Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6181280Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6181485Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6181694Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6181698Z 
2025-07-17T11:07:26.6181787Z SIGABRT(6), PID: 1817872, Thread 1818192: 
2025-07-17T11:07:26.6182259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6182571Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6182856Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6183056Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6183624Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6183902Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6184114Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6184119Z 
2025-07-17T11:07:26.6184207Z SIGABRT(6), PID: 1817868, Thread 1818012: 
2025-07-17T11:07:26.6184679Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6184884Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6185087Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6185283Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6185783Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6185989Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6186197Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6186202Z 
2025-07-17T11:07:26.6186295Z SIGABRT(6), PID: 1817870, Thread 1818240: 
2025-07-17T11:07:26.6186761Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6186971Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6187171Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6187370Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6187864Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6188082Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6188293Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6188299Z 
2025-07-17T11:07:26.6188390Z SIGABRT(6), PID: 1817872, Thread 1818195: 
2025-07-17T11:07:26.6188857Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6189067Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6189270Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6189469Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6189969Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6190179Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6190461Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6190465Z 
2025-07-17T11:07:26.6190559Z SIGABRT(6), PID: 1817870, Thread 1818244: 
2025-07-17T11:07:26.6191094Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6191309Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6191571Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6191829Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6192322Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6192530Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6192741Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6192745Z 
2025-07-17T11:07:26.6192839Z SIGABRT(6), PID: 1817868, Thread 1818016: 
2025-07-17T11:07:26.6193310Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6193518Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6193721Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6193920Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6194413Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6194622Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6194832Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6194837Z 
2025-07-17T11:07:26.6194929Z SIGABRT(6), PID: 1817870, Thread 1818248: 
2025-07-17T11:07:26.6195396Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6195607Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6195810Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6196011Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6196508Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6196722Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6196934Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6197027Z SIGABRT(6), PID: 1817872, Thread 1818199: 
2025-07-17T11:07:26.6197499Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6197706Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6197916Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6198112Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6198608Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6198890Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6199185Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6199190Z 
2025-07-17T11:07:26.6199193Z 
2025-07-17T11:07:26.6199285Z SIGABRT(6), PID: 1817868, Thread 1818020: 
2025-07-17T11:07:26.6199815Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6200081Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6200286Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6200483Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6200984Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6201192Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6201401Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6201405Z 
2025-07-17T11:07:26.6201495Z SIGABRT(6), PID: 1817870, Thread 1818252: 
2025-07-17T11:07:26.6201965Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6202174Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6202380Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6202578Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6203071Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6203278Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6203489Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6203493Z 
2025-07-17T11:07:26.6203582Z SIGABRT(6), PID: 1817867, Thread 1817867: 
2025-07-17T11:07:26.6204052Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6204256Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6204495Z frame #2: <unknown function> + 0x7741f (0x7f1261a6141f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6204725Z frame #3: <unknown function> + 0x76c8e (0x7f1261a60c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6204955Z frame #4: <unknown function> + 0x6ad69 (0x7f1261a54d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6205167Z frame #5: <unknown function> + 0x3f54db (0x7f12f26b94db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6205380Z frame #6: <unknown function> + 0x3e11d8 (0x7f12f26a51d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6205585Z frame #7: <unknown function> + 0xb2f86 (0x7f12f2376f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6205794Z frame #8: <unknown function> + 0x8e501 (0x7f12f2352501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6206175Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f131d1597ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.6206564Z frame #10: <unknown function> + 0xd307d9 (0x7f133b6a87d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6206864Z frame #11: <unknown function> + 0x139902 (0x564974970902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6207176Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x564974967a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6207409Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6207710Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x564974967a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6207992Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6208235Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6208457Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6208697Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6208917Z frame #19: <unknown function> + 0x1474b2 (0x56497497e4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6209125Z frame #20: PyObject_Call + 0xbc (0x56497497ee4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6209346Z frame #21: <unknown function> + 0x1ff69d (0x564974a3669d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6209578Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x56497496c023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6209823Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x564974967cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6210046Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6210284Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5649749633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6210508Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6210744Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5649749633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6210969Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6211205Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x564974967a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6211464Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x56497496b4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6211684Z frame #31: _PyObject_Call_Prepend + 0x69 (0x56497497cab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6211903Z frame #32: <unknown function> + 0x205949 (0x564974a3c949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6212123Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x56497496c023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6212363Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x564974967cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6212587Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6212828Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5649749633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6213048Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6213283Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5649749633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6213505Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6213742Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5649749633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6213962Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6214204Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5649749643f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6214500Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6214946Z frame #44: <unknown function> + 0x7ab3f7 (0x7f133b1233f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6215168Z frame #45: <unknown function> + 0x13bc88 (0x564974972c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6215429Z frame #46: PyObject_Call + 0x20f (0x56497497ef9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6215732Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x56497496898d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6216024Z frame #48: <unknown function> + 0x147608 (0x56497497e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6216263Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x564974965c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6216485Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6216721Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5649749633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6216944Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6217177Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5649749643f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6217404Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6217638Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5649749633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6217859Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6218093Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x564974965c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6218316Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6218553Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x564974965c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6218774Z frame #60: <unknown function> + 0x147608 (0x56497497e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6219008Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x564974965c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6224236Z frame #62: <unknown function> + 0x147608 (0x56497497e608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6224366Z SIGABRT(6), PID: 1817872, Thread 1818201: 
2025-07-17T11:07:26.6224889Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6225123Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6225338Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6225551Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6226078Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6226297Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6226517Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6226524Z 
2025-07-17T11:07:26.6226528Z 
2025-07-17T11:07:26.6226627Z SIGABRT(6), PID: 1817868, Thread 1818023: 
2025-07-17T11:07:26.6227125Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6227342Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6227694Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6227980Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6228497Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6228782Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6229068Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6229073Z 
2025-07-17T11:07:26.6229166Z SIGABRT(6), PID: 1817870, Thread 1818256: 
2025-07-17T11:07:26.6229644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6229854Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6230055Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6230258Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6230770Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6230977Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6231188Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6231193Z 
2025-07-17T11:07:26.6231284Z SIGABRT(6), PID: 1817872, Thread 1818204: 
2025-07-17T11:07:26.6231763Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6231978Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6232183Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6232385Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6232889Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6233100Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6233308Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6233312Z 
2025-07-17T11:07:26.6233404Z SIGABRT(6), PID: 1817868, Thread 1818026: 
2025-07-17T11:07:26.6233870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6234078Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6234280Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6234475Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6234970Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6235177Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6235383Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6235388Z 
2025-07-17T11:07:26.6235578Z SIGABRT(6), PID: 1817870, Thread 1818260: 
2025-07-17T11:07:26.6236040Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6236310Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6236512Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6236766Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6237320Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6237529Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6237736Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6237829Z SIGABRT(6), PID: 1817867, Thread 1817917: 
2025-07-17T11:07:26.6238297Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6238504Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6238705Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6238901Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6239390Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6239595Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6239797Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6239806Z 
2025-07-17T11:07:26.6239809Z 
2025-07-17T11:07:26.6239896Z SIGABRT(6), PID: 1817872, Thread 1818208: 
2025-07-17T11:07:26.6240364Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6240569Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6240777Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6240972Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6241463Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6241668Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6241884Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6241888Z 
2025-07-17T11:07:26.6241976Z SIGABRT(6), PID: 1817868, Thread 1818029: 
2025-07-17T11:07:26.6242448Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6242655Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6242858Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6243053Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6243548Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6243831Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6244040Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6244106Z 
2025-07-17T11:07:26.6244196Z SIGABRT(6), PID: 1817870, Thread 1818264: 
2025-07-17T11:07:26.6244725Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6244931Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6245199Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6245394Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6245891Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6246098Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6246309Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6246313Z 
2025-07-17T11:07:26.6246399Z SIGABRT(6), PID: 1817867, Thread 1817920: 
2025-07-17T11:07:26.6246863Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6247071Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6247276Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6247468Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6247957Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6248165Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6248372Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6248377Z 
2025-07-17T11:07:26.6248462Z SIGABRT(6), PID: 1817868, Thread 1818032: 
2025-07-17T11:07:26.6248927Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6249132Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6249332Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6249528Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6250018Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6250225Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6250429Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6250433Z 
2025-07-17T11:07:26.6250525Z SIGABRT(6), PID: 1817872, Thread 1818213: 
2025-07-17T11:07:26.6250986Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6251194Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6251395Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6251670Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6252217Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6252427Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6252694Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6252699Z 
2025-07-17T11:07:26.6252790Z SIGABRT(6), PID: 1817870, Thread 1818268: 
2025-07-17T11:07:26.6253315Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6253524Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6253723Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6253920Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6254413Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6254620Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6254826Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6254832Z 
2025-07-17T11:07:26.6254919Z SIGABRT(6), PID: 1817867, Thread 1817924: 
2025-07-17T11:07:26.6255378Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6255582Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6255786Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6256095Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6256588Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6256796Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6257001Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6257005Z 
2025-07-17T11:07:26.6257094Z SIGABRT(6), PID: 1817868, Thread 1818035: 
2025-07-17T11:07:26.6257556Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6257764Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6257965Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6258163Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6258654Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6258861Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6259066Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6259073Z 
2025-07-17T11:07:26.6259160Z SIGABRT(6), PID: 1817872, Thread 1818217: 
2025-07-17T11:07:26.6259624Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6259933Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6260212Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6260411Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6261095Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6261416Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6261626Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6261631Z 
2025-07-17T11:07:26.6261718Z SIGABRT(6), PID: 1817870, Thread 1818272: 
2025-07-17T11:07:26.6262192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6262401Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6262603Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6262797Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6263298Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6263501Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6263709Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6263713Z 
2025-07-17T11:07:26.6263799Z SIGABRT(6), PID: 1817868, Thread 1818038: 
2025-07-17T11:07:26.6264267Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6264471Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6264671Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6264865Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6265353Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6265554Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6265760Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6265767Z 
2025-07-17T11:07:26.6265853Z SIGABRT(6), PID: 1817867, Thread 1817927: 
2025-07-17T11:07:26.6266317Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6266518Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6266722Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6266915Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6267404Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6267610Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6267815Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6267892Z 
2025-07-17T11:07:26.6267982Z SIGABRT(6), PID: 1817872, Thread 1818221: 
2025-07-17T11:07:26.6268511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6268720Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6268986Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6269243Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6269734Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6269939Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6270148Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6270151Z 
2025-07-17T11:07:26.6270237Z SIGABRT(6), PID: 1817870, Thread 1818276: 
2025-07-17T11:07:26.6270699Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6270907Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6271108Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6271304Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6271797Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6272003Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6272207Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6272213Z 
2025-07-17T11:07:26.6272302Z SIGABRT(6), PID: 1817868, Thread 1818041: 
2025-07-17T11:07:26.6272762Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6272966Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6273168Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6273364Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6273850Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6274058Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6274266Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6274271Z 
2025-07-17T11:07:26.6274359Z SIGABRT(6), PID: 1817867, Thread 1817931: 
2025-07-17T11:07:26.6274818Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6275024Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6275222Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6275418Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6275907Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6276248Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6276458Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6276547Z SIGABRT(6), PID: 1817872, Thread 1818225: 
2025-07-17T11:07:26.6277068Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6277334Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6277535Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6277734Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6278225Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6278433Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6278637Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6278641Z 
2025-07-17T11:07:26.6278645Z 
2025-07-17T11:07:26.6278734Z SIGABRT(6), PID: 1817868, Thread 1818044: 
2025-07-17T11:07:26.6279192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6279397Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6279594Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6279792Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6280283Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6280487Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6280694Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6280700Z 
2025-07-17T11:07:26.6280786Z SIGABRT(6), PID: 1817870, Thread 1818280: 
2025-07-17T11:07:26.6281253Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6281457Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6281660Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6281858Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6282353Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6282556Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6282763Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6282768Z 
2025-07-17T11:07:26.6282854Z SIGABRT(6), PID: 1817867, Thread 1817936: 
2025-07-17T11:07:26.6283317Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6283520Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6283797Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6283991Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6284544Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6284810Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6285022Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6285087Z 
2025-07-17T11:07:26.6285176Z SIGABRT(6), PID: 1817868, Thread 1818047: 
2025-07-17T11:07:26.6285644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6285847Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6286049Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6286245Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6286735Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6286938Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6287145Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6287149Z 
2025-07-17T11:07:26.6287234Z SIGABRT(6), PID: 1817870, Thread 1818284: 
2025-07-17T11:07:26.6287695Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6287899Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6288105Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6288297Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6288793Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6288997Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6289204Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6289208Z 
2025-07-17T11:07:26.6289294Z SIGABRT(6), PID: 1817867, Thread 1817940: 
2025-07-17T11:07:26.6289756Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6289960Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6290168Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6290360Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6290850Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6291054Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6291261Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6291265Z 
2025-07-17T11:07:26.6291355Z SIGABRT(6), PID: 1817868, Thread 1818051: 
2025-07-17T11:07:26.6291898Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6292162Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6292366Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6292641Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6293136Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6293403Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6293613Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6293619Z 
2025-07-17T11:07:26.6293709Z SIGABRT(6), PID: 1817870, Thread 1818288: 
2025-07-17T11:07:26.6294179Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6294388Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6294589Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6294786Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6295294Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6295514Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6295723Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6295728Z 
2025-07-17T11:07:26.6295820Z SIGABRT(6), PID: 1817872, Thread 1818229: 
2025-07-17T11:07:26.6296365Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6296578Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6296783Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6296985Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6297490Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6297702Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6297911Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6297916Z 
2025-07-17T11:07:26.6298007Z SIGABRT(6), PID: 1817867, Thread 1817944: 
2025-07-17T11:07:26.6298475Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6298683Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6298880Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6299077Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6299565Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6299867Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6300072Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6300077Z 
2025-07-17T11:07:26.6300240Z SIGABRT(6), PID: 1817868, Thread 1818054: 
2025-07-17T11:07:26.6300708Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6300985Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6301259Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6301460Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6301951Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6302160Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6302366Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6302456Z SIGABRT(6), PID: 1817870, Thread 1818292: 
2025-07-17T11:07:26.6302920Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6303121Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6303324Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6303516Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6304011Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6304220Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6304428Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6304433Z 
2025-07-17T11:07:26.6304436Z 
2025-07-17T11:07:26.6304522Z SIGABRT(6), PID: 1817872, Thread 1818233: 
2025-07-17T11:07:26.6304989Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6305199Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6305406Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6305603Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6306105Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6306313Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6306521Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6306525Z 
2025-07-17T11:07:26.6306615Z SIGABRT(6), PID: 1817867, Thread 1817948: 
2025-07-17T11:07:26.6307086Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6307291Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6307494Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6307688Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6308324Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6308531Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6308737Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6308809Z 
2025-07-17T11:07:26.6308901Z SIGABRT(6), PID: 1817870, Thread 1818297: 
2025-07-17T11:07:26.6309428Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6309630Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6309832Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6310026Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6310525Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6310743Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6310953Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6310958Z 
2025-07-17T11:07:26.6311045Z SIGABRT(6), PID: 1817868, Thread 1818057: 
2025-07-17T11:07:26.6311513Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6311715Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6311917Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6312109Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6312601Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6312806Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6313010Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6313015Z 
2025-07-17T11:07:26.6313103Z SIGABRT(6), PID: 1817872, Thread 1818237: 
2025-07-17T11:07:26.6313562Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6313768Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6313973Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6314169Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6314657Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6314863Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6315070Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6315073Z 
2025-07-17T11:07:26.6315164Z SIGABRT(6), PID: 1817870, Thread 1818302: 
2025-07-17T11:07:26.6315626Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6315905Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6316175Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6316377Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6316927Z frame #4: <unknown function> + 0x3475ab (0x7f6bb10ff5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6317141Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6317411Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6317501Z SIGABRT(6), PID: 1817867, Thread 1817952: 
2025-07-17T11:07:26.6317964Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6318173Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6318376Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6318569Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6319056Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6319263Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6319474Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6319478Z 
2025-07-17T11:07:26.6319481Z 
2025-07-17T11:07:26.6319572Z SIGABRT(6), PID: 1817868, Thread 1818061: 
2025-07-17T11:07:26.6320038Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6320246Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6320445Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6320643Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6321130Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6321336Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6321543Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6321547Z 
2025-07-17T11:07:26.6321640Z SIGABRT(6), PID: 1817872, Thread 1818241: 
2025-07-17T11:07:26.6322100Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6322306Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6322509Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6322706Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6323195Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6323401Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6323610Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6323683Z 
2025-07-17T11:07:26.6323776Z SIGABRT(6), PID: 1817870, Thread 1818386: 
2025-07-17T11:07:26.6324324Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6324779Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f6ac295d08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6325044Z frame #2: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6325291Z frame #3: pthread_kill + 0x12c (0x7f6bb42699fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6325455Z frame #4: raise + 0x16 (0x7f6bb4215476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6325613Z frame #5: abort + 0xd3 (0x7f6bb41fb7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6325853Z frame #6: <unknown function> + 0x972cc (0x7f6a304e62cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6326082Z frame #7: <unknown function> + 0x95f47 (0x7f6a304e4f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6326306Z frame #8: <unknown function> + 0x34b71 (0x7f6a30483b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6326512Z frame #9: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6326728Z frame #10: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6326821Z SIGABRT(6), PID: 1817868, Thread 1818064: 
2025-07-17T11:07:26.6327289Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6327492Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6327696Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6327894Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6328391Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6328604Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6328815Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6328905Z SIGABRT(6), PID: 1817867, Thread 1817955: 
2025-07-17T11:07:26.6329375Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6329577Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6329779Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6329975Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6330467Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6330669Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6330878Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6330967Z SIGABRT(6), PID: 1817872, Thread 1818245: 
2025-07-17T11:07:26.6331431Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6331634Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6331913Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6332108Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6332661Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6332930Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6333143Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6333206Z 
2025-07-17T11:07:26.6333209Z 
2025-07-17T11:07:26.6333213Z 
2025-07-17T11:07:26.6333216Z 
2025-07-17T11:07:26.6333304Z SIGABRT(6), PID: 1817872, Thread 1818249: 
2025-07-17T11:07:26.6333771Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6333975Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6334181Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6334377Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6334872Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6335081Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6335285Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6335289Z 
2025-07-17T11:07:26.6335379Z SIGABRT(6), PID: 1817868, Thread 1818067: 
2025-07-17T11:07:26.6335834Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6336102Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6336303Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6336498Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6336991Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6337198Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6337401Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6337405Z 
2025-07-17T11:07:26.6337496Z SIGABRT(6), PID: 1817867, Thread 1817960: 
2025-07-17T11:07:26.6337961Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6338166Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6338376Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6338575Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6339063Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6339270Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6339472Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6339568Z 
2025-07-17T11:07:26.6339662Z SIGABRT(6), PID: 1817870, Thread 1821110: 
2025-07-17T11:07:26.6340201Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6340410Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6340580Z frame #2: __poll + 0x4f (0x7f6bb42ebbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6341088Z frame #3: <unknown function> + 0x489f (0x7f6bb3df789f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.6341390Z frame #4: <unknown function> + 0x1483bc (0x563c6e1d13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6341638Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6341864Z frame #6: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6342101Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6342323Z frame #8: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6342567Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x563c6e1b9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6342793Z frame #10: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6343038Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x563c6e1b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6343265Z frame #12: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6343501Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6343719Z frame #14: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6343957Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6344175Z frame #16: <unknown function> + 0x147734 (0x563c6e1d0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6344394Z frame #17: <unknown function> + 0x21ca82 (0x563c6e2a5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6344611Z frame #18: <unknown function> + 0x21c9e4 (0x563c6e2a59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6344821Z frame #19: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6345029Z frame #20: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6345033Z 
2025-07-17T11:07:26.6345124Z SIGABRT(6), PID: 1817872, Thread 1818253: 
2025-07-17T11:07:26.6345591Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6345805Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6346009Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6346209Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6346705Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6346914Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6347118Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6347123Z 
2025-07-17T11:07:26.6347212Z SIGABRT(6), PID: 1817868, Thread 1818070: 
2025-07-17T11:07:26.6347678Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6347959Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6348217Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6348418Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6348970Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6349238Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6349441Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6349447Z 
2025-07-17T11:07:26.6349534Z SIGABRT(6), PID: 1817867, Thread 1817964: 
2025-07-17T11:07:26.6350008Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6350211Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6350414Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6350608Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6351102Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6351305Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6351510Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6351515Z 
2025-07-17T11:07:26.6351601Z SIGABRT(6), PID: 1817872, Thread 1818257: 
2025-07-17T11:07:26.6352071Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6352275Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6352481Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6352675Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6353167Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6353372Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6353581Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6353587Z 
2025-07-17T11:07:26.6353674Z SIGABRT(6), PID: 1817868, Thread 1818073: 
2025-07-17T11:07:26.6354138Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6354340Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6354542Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6354734Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6355227Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6355427Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6355729Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6355733Z 
2025-07-17T11:07:26.6355820Z SIGABRT(6), PID: 1817867, Thread 1817968: 
2025-07-17T11:07:26.6356343Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6356608Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6356813Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6357064Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6357560Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6357763Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6357969Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6357973Z 
2025-07-17T11:07:26.6358060Z SIGABRT(6), PID: 1817872, Thread 1818261: 
2025-07-17T11:07:26.6358527Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6358732Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6358937Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6359130Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6359622Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6359828Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6360038Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6360042Z 
2025-07-17T11:07:26.6360132Z SIGABRT(6), PID: 1817870, Thread 1822116: 
2025-07-17T11:07:26.6360594Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6360799Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6361000Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6361222Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6bb42672dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6361664Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f6aee6a5048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6362090Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f6aee6a7bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6362321Z frame #6: <unknown function> + 0xdc253 (0x7f6ac0bc9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6362528Z frame #7: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6362732Z frame #8: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6362738Z 
2025-07-17T11:07:26.6362827Z SIGABRT(6), PID: 1817868, Thread 1818076: 
2025-07-17T11:07:26.6363293Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6363499Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6363772Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6364031Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6364526Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6364790Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6365054Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6365059Z 
2025-07-17T11:07:26.6365150Z SIGABRT(6), PID: 1817867, Thread 1817973: 
2025-07-17T11:07:26.6365610Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6365820Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6366019Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6366217Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6366709Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6366915Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6367120Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6367124Z 
2025-07-17T11:07:26.6367212Z SIGABRT(6), PID: 1817872, Thread 1818265: 
2025-07-17T11:07:26.6367671Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6367877Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6368080Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6368278Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6368771Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6368978Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6369182Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6369186Z 
2025-07-17T11:07:26.6369275Z SIGABRT(6), PID: 1817868, Thread 1818080: 
2025-07-17T11:07:26.6369737Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6369946Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6370144Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6370339Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6370831Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6371037Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6371245Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6371249Z 
2025-07-17T11:07:26.6371409Z SIGABRT(6), PID: 1817867, Thread 1817977: 
2025-07-17T11:07:26.6371876Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6372139Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6372345Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6372597Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6373147Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6373354Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6373559Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6373565Z 
2025-07-17T11:07:26.6373650Z SIGABRT(6), PID: 1817872, Thread 1818269: 
2025-07-17T11:07:26.6374116Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6374320Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6374525Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6374718Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6375211Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6375415Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6375623Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6375627Z 
2025-07-17T11:07:26.6375713Z SIGABRT(6), PID: 1817870, Thread 1822117: 
2025-07-17T11:07:26.6376244Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6376447Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6376650Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6376870Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6bb42672dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6377341Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f6aee69f741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6377568Z frame #5: <unknown function> + 0xdc253 (0x7f6ac0bc9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6377775Z frame #6: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6377981Z frame #7: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6377985Z 
2025-07-17T11:07:26.6378077Z SIGABRT(6), PID: 1817868, Thread 1818083: 
2025-07-17T11:07:26.6378536Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6378742Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6378940Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6379133Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6379625Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6379920Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6380199Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6380204Z 
2025-07-17T11:07:26.6380296Z SIGABRT(6), PID: 1817867, Thread 1817980: 
2025-07-17T11:07:26.6380835Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6381134Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6381334Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6381529Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6382027Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6382231Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6382436Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6382525Z SIGABRT(6), PID: 1817872, Thread 1818273: 
2025-07-17T11:07:26.6382996Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6383201Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6383406Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6383601Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6384091Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6384297Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6384502Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6384508Z 
2025-07-17T11:07:26.6384511Z 
2025-07-17T11:07:26.6384597Z SIGABRT(6), PID: 1817870, Thread 1822118: 
2025-07-17T11:07:26.6385063Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6385264Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6385433Z frame #2: ioctl + 0x3f (0x7f6bb42ed94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6385667Z frame #3: <unknown function> + 0x13eef0 (0x7f6a3058def0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6385899Z frame #4: <unknown function> + 0x1378c4 (0x7f6a305868c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6386125Z frame #5: <unknown function> + 0x95b2b (0x7f6a304e4b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6386350Z frame #6: <unknown function> + 0x34b71 (0x7f6a30483b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6386555Z frame #7: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6386763Z frame #8: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6386849Z SIGABRT(6), PID: 1817868, Thread 1818086: 
2025-07-17T11:07:26.6387317Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6387596Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6387797Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6388164Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6388722Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6388929Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6389196Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6389283Z SIGABRT(6), PID: 1817872, Thread 1818277: 
2025-07-17T11:07:26.6389751Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6389957Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6390163Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6390357Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6390853Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6391057Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6391266Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6391352Z SIGABRT(6), PID: 1817867, Thread 1817985: 
2025-07-17T11:07:26.6391818Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6392022Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6392229Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6392420Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6392912Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6393114Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6393319Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6393323Z 
2025-07-17T11:07:26.6393326Z 
2025-07-17T11:07:26.6393329Z 
2025-07-17T11:07:26.6393334Z 
2025-07-17T11:07:26.6393422Z SIGABRT(6), PID: 1817872, Thread 1818281: 
2025-07-17T11:07:26.6393885Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6394091Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6394292Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6394492Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6394983Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6395191Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6395394Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6395470Z 
2025-07-17T11:07:26.6395560Z SIGABRT(6), PID: 1817868, Thread 1818089: 
2025-07-17T11:07:26.6396085Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6396294Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6396550Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6396752Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6397330Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6397543Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6397756Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6397760Z 
2025-07-17T11:07:26.6397850Z SIGABRT(6), PID: 1817867, Thread 1817991: 
2025-07-17T11:07:26.6398324Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6398539Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6398740Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6398937Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6399434Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6399639Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6399844Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6399935Z SIGABRT(6), PID: 1817870, Thread 1822140: 
2025-07-17T11:07:26.6400404Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6400612Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6400777Z frame #2: __poll + 0x4f (0x7f6bb42ebbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6400993Z frame #3: <unknown function> + 0xd4540 (0x7f6a786fd540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.6401195Z frame #4: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6401404Z frame #5: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6401411Z 
2025-07-17T11:07:26.6401497Z SIGABRT(6), PID: 1817872, Thread 1818285: 
2025-07-17T11:07:26.6401965Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6402170Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6402376Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6402575Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6403073Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6403276Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6403566Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6403571Z 
2025-07-17T11:07:26.6403574Z 
2025-07-17T11:07:26.6403663Z SIGABRT(6), PID: 1817868, Thread 1818092: 
2025-07-17T11:07:26.6404192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6404461Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6404667Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6404929Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6405420Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6405629Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6405832Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6405836Z 
2025-07-17T11:07:26.6405927Z SIGABRT(6), PID: 1817870, Thread 1822148: 
2025-07-17T11:07:26.6406392Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6406598Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6406762Z frame #2: __poll + 0x4f (0x7f6bb42ebbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6406977Z frame #3: <unknown function> + 0xd65a2 (0x7f6a786ff5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.6407179Z frame #4: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6407388Z frame #5: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6407394Z 
2025-07-17T11:07:26.6407483Z SIGABRT(6), PID: 1817872, Thread 1818289: 
2025-07-17T11:07:26.6407951Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6408156Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6408363Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6408561Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6409057Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6409261Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6409470Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6409475Z 
2025-07-17T11:07:26.6409560Z SIGABRT(6), PID: 1817867, Thread 1817995: 
2025-07-17T11:07:26.6410029Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6410232Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6410434Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6410628Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6411118Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6411397Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6411604Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6411669Z 
2025-07-17T11:07:26.6411758Z SIGABRT(6), PID: 1817870, Thread 1822157: 
2025-07-17T11:07:26.6412302Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6412508Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6412773Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6412971Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6413197Z frame #4: <unknown function> + 0xd1e7f (0x7f6a786fae7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.6413414Z frame #5: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6413627Z frame #6: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6413631Z 
2025-07-17T11:07:26.6413728Z SIGABRT(6), PID: 1817868, Thread 1818095: 
2025-07-17T11:07:26.6414204Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6414415Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6414620Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6414819Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6415327Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6415536Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6415748Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6415753Z 
2025-07-17T11:07:26.6415847Z SIGABRT(6), PID: 1817872, Thread 1818293: 
2025-07-17T11:07:26.6416367Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6416579Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6416781Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6416982Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6417477Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6417689Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6417897Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6417990Z SIGABRT(6), PID: 1817867, Thread 1817999: 
2025-07-17T11:07:26.6418454Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6418666Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6418870Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6419069Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6419649Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6419948Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6420159Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6420163Z 
2025-07-17T11:07:26.6420166Z 
2025-07-17T11:07:26.6420331Z SIGABRT(6), PID: 1817868, Thread 1818098: 
2025-07-17T11:07:26.6420802Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6421088Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6421292Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6421494Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6421988Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6422195Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6422402Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6422497Z SIGABRT(6), PID: 1817870, Thread 1822169: 
2025-07-17T11:07:26.6422964Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6423171Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6423374Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6423588Z frame #3: <unknown function> + 0x9ca53 (0x7f6bb426fa53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6423847Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x563c6e19dbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6424080Z frame #5: <unknown function> + 0x1d314f (0x563c6e25c14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6424303Z frame #6: <unknown function> + 0x1d2f6b (0x563c6e25bf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6424527Z frame #7: <unknown function> + 0x6df36 (0x563c6e0f6f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6424772Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6424999Z frame #9: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6425240Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6425476Z frame #11: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6425710Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6425938Z frame #13: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6426172Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6426399Z frame #15: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6426642Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6426861Z frame #17: <unknown function> + 0x147734 (0x563c6e1d0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6427082Z frame #18: <unknown function> + 0x21ca82 (0x563c6e2a5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6427381Z frame #19: <unknown function> + 0x21c9e4 (0x563c6e2a59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6427593Z frame #20: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6427870Z frame #21: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6427875Z 
2025-07-17T11:07:26.6427878Z 
2025-07-17T11:07:26.6427975Z SIGABRT(6), PID: 1817872, Thread 1818296: 
2025-07-17T11:07:26.6428510Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6428782Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6428997Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6429196Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6429696Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6429906Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6430118Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6430208Z SIGABRT(6), PID: 1817867, Thread 1818003: 
2025-07-17T11:07:26.6430677Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6430882Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6431088Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6431287Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6431783Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6431993Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6432199Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6432204Z 
2025-07-17T11:07:26.6432207Z 
2025-07-17T11:07:26.6432299Z SIGABRT(6), PID: 1817868, Thread 1818102: 
2025-07-17T11:07:26.6432765Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6432970Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6433173Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6433376Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6433875Z frame #4: <unknown function> + 0x3475ab (0x7f6a8f5135ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6434086Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6434296Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6434302Z 
2025-07-17T11:07:26.6434392Z SIGABRT(6), PID: 1817870, Thread 1822186: 
2025-07-17T11:07:26.6434859Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6435067Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6435311Z frame #2: read + 0x4c (0x7f6bb42e781c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6435498Z frame #3: _Py_read + 0x4c (0x563c6e263a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6435785Z frame #4: <unknown function> + 0x208c3b (0x563c6e291c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6436014Z frame #5: <unknown function> + 0x14648d (0x563c6e1cf48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6436322Z frame #6: PyObject_VectorcallMethod + 0x85 (0x563c6e1de215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6436554Z frame #7: <unknown function> + 0x208ae7 (0x563c6e291ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6436833Z frame #8: <unknown function> + 0x208472 (0x563c6e291472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6437055Z frame #9: <unknown function> + 0x1da6ee (0x563c6e2636ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6437277Z frame #10: <unknown function> + 0x1483bc (0x563c6e1d13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6437524Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6437753Z frame #12: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6437991Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x563c6e1b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6438211Z frame #14: <unknown function> + 0x147734 (0x563c6e1d0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6438461Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x563c6e1b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6438688Z frame #16: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6438923Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6439149Z frame #18: _PyFunction_Vectorcall + 0x6c (0x563c6e1c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6439386Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x563c6e1b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6439607Z frame #20: <unknown function> + 0x147734 (0x563c6e1d0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6439827Z frame #21: <unknown function> + 0x21ca82 (0x563c6e2a5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6440045Z frame #22: <unknown function> + 0x21c9e4 (0x563c6e2a59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6440266Z frame #23: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6440479Z frame #24: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6440483Z 
2025-07-17T11:07:26.6440575Z SIGABRT(6), PID: 1817872, Thread 1818301: 
2025-07-17T11:07:26.6441049Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6441266Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6441474Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6441681Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6442182Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6442394Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6442604Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6442696Z SIGABRT(6), PID: 1817867, Thread 1818007: 
2025-07-17T11:07:26.6443163Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6443445Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6443731Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6443933Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6444488Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6444757Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6444967Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6444971Z 
2025-07-17T11:07:26.6444974Z 
2025-07-17T11:07:26.6445068Z SIGABRT(6), PID: 1817868, Thread 1818388: 
2025-07-17T11:07:26.6445543Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6445996Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f69a0d7208a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6446203Z frame #2: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6446400Z frame #3: pthread_kill + 0x12c (0x7f6a9267d9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6446568Z frame #4: raise + 0x16 (0x7f6a92629476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6446729Z frame #5: abort + 0xd3 (0x7f6a9260f7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6446967Z frame #6: <unknown function> + 0x972cc (0x7f690e8fb2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6447197Z frame #7: <unknown function> + 0x95f47 (0x7f690e8f9f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6447422Z frame #8: <unknown function> + 0x34b71 (0x7f690e898b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6447635Z frame #9: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6447852Z frame #10: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6447859Z 
2025-07-17T11:07:26.6447950Z SIGABRT(6), PID: 1817867, Thread 1818010: 
2025-07-17T11:07:26.6448429Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6448635Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6448841Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6449037Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6449544Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6449748Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6449961Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6449967Z 
2025-07-17T11:07:26.6450058Z SIGABRT(6), PID: 1817872, Thread 1818305: 
2025-07-17T11:07:26.6450530Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6450734Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6450939Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6451214Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6451776Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6451987Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6452259Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6452264Z 
2025-07-17T11:07:26.6452415Z SIGABRT(6), PID: 1817868, Thread 1822124: 
2025-07-17T11:07:26.6452885Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6453091Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6453261Z frame #2: __poll + 0x4f (0x7f6a926ffbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6453670Z frame #3: <unknown function> + 0x489f (0x7f6a9220b89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.6453897Z frame #4: <unknown function> + 0x1483bc (0x559370a473bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6454138Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6454368Z frame #6: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6454608Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6454835Z frame #8: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6455081Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x559370a2fa65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6455315Z frame #10: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6455562Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x559370a2dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6455787Z frame #12: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6456099Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6456326Z frame #14: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6456562Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6456783Z frame #16: <unknown function> + 0x147734 (0x559370a46734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6457001Z frame #17: <unknown function> + 0x21ca82 (0x559370b1ba82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6457222Z frame #18: <unknown function> + 0x21c9e4 (0x559370b1b9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6457436Z frame #19: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6457651Z frame #20: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6457743Z SIGABRT(6), PID: 1817867, Thread 1818013: 
2025-07-17T11:07:26.6458218Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6458431Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6458639Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6458837Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6459342Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6459717Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6459950Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6460039Z SIGABRT(6), PID: 1817872, Thread 1818308: 
2025-07-17T11:07:26.6460581Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6460859Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6461067Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6461267Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6461769Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6461978Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6462191Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6462195Z 
2025-07-17T11:07:26.6462198Z 
2025-07-17T11:07:26.6462203Z 
2025-07-17T11:07:26.6462295Z SIGABRT(6), PID: 1817867, Thread 1818015: 
2025-07-17T11:07:26.6462766Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6462971Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6463182Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6463377Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6463875Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6464085Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6464293Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6464382Z SIGABRT(6), PID: 1817872, Thread 1818311: 
2025-07-17T11:07:26.6464850Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6465063Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6465266Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6465467Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6465959Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6466171Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6466380Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6466385Z 
2025-07-17T11:07:26.6466388Z 
2025-07-17T11:07:26.6466479Z SIGABRT(6), PID: 1817867, Thread 1818018: 
2025-07-17T11:07:26.6466943Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6467149Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6467427Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6467683Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6468179Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6468445Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6468714Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6468719Z 
2025-07-17T11:07:26.6468810Z SIGABRT(6), PID: 1817872, Thread 1818314: 
2025-07-17T11:07:26.6469285Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6469500Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6469706Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6469907Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6470419Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6470631Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6470843Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6470847Z 
2025-07-17T11:07:26.6470940Z SIGABRT(6), PID: 1817867, Thread 1818021: 
2025-07-17T11:07:26.6471415Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6471624Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6471828Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6472031Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6472528Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6472740Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6472947Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6473043Z SIGABRT(6), PID: 1817868, Thread 1822125: 
2025-07-17T11:07:26.6473508Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6473718Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6473922Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6474146Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6a9267b2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6474592Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f69ccaba048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6475023Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f69ccabcbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6475258Z frame #6: <unknown function> + 0xdc253 (0x7f699efde253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6475560Z frame #7: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6475770Z frame #8: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6475775Z 
2025-07-17T11:07:26.6475924Z SIGABRT(6), PID: 1817872, Thread 1818317: 
2025-07-17T11:07:26.6476400Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6476666Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6476936Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6477141Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6477638Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6477847Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6478061Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6478065Z 
2025-07-17T11:07:26.6478069Z 
2025-07-17T11:07:26.6478158Z SIGABRT(6), PID: 1817870, Thread 1822236: 
2025-07-17T11:07:26.6478639Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6478846Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6479050Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6479248Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6479631Z frame #4: <unknown function> + 0x5a02023 (0x7f6af8693023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6480201Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6af86971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6480804Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6af868e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6481189Z frame #7: <unknown function> + 0x79d655 (0x7f6b09b7e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6481426Z frame #8: <unknown function> + 0xdc253 (0x7f6ac0bc9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6481632Z frame #9: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6481854Z frame #10: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6481858Z 
2025-07-17T11:07:26.6481948Z SIGABRT(6), PID: 1817867, Thread 1818024: 
2025-07-17T11:07:26.6482424Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6482631Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6482835Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6483033Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6483538Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6483743Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6484026Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6484030Z 
2025-07-17T11:07:26.6484177Z SIGABRT(6), PID: 1817872, Thread 1818320: 
2025-07-17T11:07:26.6484651Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6484917Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6485180Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6485380Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6485878Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6486090Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6486299Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6486304Z 
2025-07-17T11:07:26.6486395Z SIGABRT(6), PID: 1817867, Thread 1818028: 
2025-07-17T11:07:26.6486867Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6487075Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6487280Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6487474Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6487969Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6488177Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6488391Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6488483Z SIGABRT(6), PID: 1817868, Thread 1822126: 
2025-07-17T11:07:26.6488949Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6489158Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6489358Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6489582Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6a9267b2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6490051Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f69ccab4741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6490289Z frame #5: <unknown function> + 0xdc253 (0x7f699efde253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6490493Z frame #6: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6490703Z frame #7: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6490709Z 
2025-07-17T11:07:26.6490713Z 
2025-07-17T11:07:26.6490800Z SIGABRT(6), PID: 1817872, Thread 1818322: 
2025-07-17T11:07:26.6491274Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6491480Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6491685Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6491957Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6492518Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6492732Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6493003Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6493062Z 
2025-07-17T11:07:26.6493154Z SIGABRT(6), PID: 1817868, Thread 1822129: 
2025-07-17T11:07:26.6493625Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6493831Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6494007Z frame #2: ioctl + 0x3f (0x7f6a9270194f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6494245Z frame #3: <unknown function> + 0x13eef0 (0x7f690e9a2ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6494481Z frame #4: <unknown function> + 0x1378c4 (0x7f690e99b8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6494710Z frame #5: <unknown function> + 0x95b2b (0x7f690e8f9b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6494940Z frame #6: <unknown function> + 0x34b71 (0x7f690e898b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6495149Z frame #7: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6495360Z frame #8: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6495363Z 
2025-07-17T11:07:26.6495453Z SIGABRT(6), PID: 1817873, Thread 1817873: 
2025-07-17T11:07:26.6495987Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6496200Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6496425Z frame #2: <unknown function> + 0x77435 (0x7f47de363435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6496655Z frame #3: <unknown function> + 0x76c8e (0x7f47de362c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6496880Z frame #4: <unknown function> + 0x6ad69 (0x7f47de356d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6497096Z frame #5: <unknown function> + 0x3f54db (0x7f486efbb4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6497310Z frame #6: <unknown function> + 0x3e11d8 (0x7f486efa71d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6497518Z frame #7: <unknown function> + 0xb2f86 (0x7f486ec78f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6497725Z frame #8: <unknown function> + 0x8e501 (0x7f486ec54501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6498110Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f4899a5b7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.6498500Z frame #10: <unknown function> + 0xd307d9 (0x7f48b7fae7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6498731Z frame #11: <unknown function> + 0x139902 (0x55b7d73dd902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6498986Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55b7d73d4a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6499217Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6499465Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55b7d73d4a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6499691Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6500043Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6500345Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6500583Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6500878Z frame #19: <unknown function> + 0x1474b2 (0x55b7d73eb4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6501086Z frame #20: PyObject_Call + 0xbc (0x55b7d73ebe4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6501402Z frame #21: <unknown function> + 0x1ff69d (0x55b7d74a369d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6501630Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55b7d73d9023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6501873Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55b7d73d4cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6502099Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6502336Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55b7d73d03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6502562Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6502794Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55b7d73d03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6503020Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6503262Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55b7d73d4a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6503517Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55b7d73d84b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6503738Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55b7d73e9ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6503962Z frame #32: <unknown function> + 0x205949 (0x55b7d74a9949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6504186Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55b7d73d9023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6504433Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55b7d73d4cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6504656Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6504896Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55b7d73d03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6505120Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6505358Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55b7d73d03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6505579Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6505816Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55b7d73d03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6506038Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6506281Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55b7d73d13f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6506502Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6506898Z frame #44: <unknown function> + 0x7ab3f7 (0x7f48b7a293f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6507118Z frame #45: <unknown function> + 0x13bc88 (0x55b7d73dfc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6507323Z frame #46: PyObject_Call + 0x20f (0x55b7d73ebf9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6507568Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55b7d73d598d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6507869Z frame #48: <unknown function> + 0x147608 (0x55b7d73eb608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6508169Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b7d73d2c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6508400Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6508638Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55b7d73d03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6508925Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6509223Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55b7d73d13f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6509447Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6509683Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55b7d73d03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6509909Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6510149Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b7d73d2c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6510374Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6510610Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b7d73d2c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6510831Z frame #60: <unknown function> + 0x147608 (0x55b7d73eb608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6511070Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b7d73d2c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6511292Z frame #62: <unknown function> + 0x147608 (0x55b7d73eb608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6511297Z 
2025-07-17T11:07:26.6511397Z SIGABRT(6), PID: 1817867, Thread 1818031: 
2025-07-17T11:07:26.6511884Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6512105Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6512315Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6512517Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6513020Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6513230Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6513439Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6513443Z 
2025-07-17T11:07:26.6513538Z SIGABRT(6), PID: 1817872, Thread 1818323: 
2025-07-17T11:07:26.6514006Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6514219Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6514424Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6514633Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6515135Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6515345Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6515554Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6515836Z SIGABRT(6), PID: 1817867, Thread 1818034: 
2025-07-17T11:07:26.6516369Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6516581Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6516841Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6517045Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6517606Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6517813Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6518020Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6518027Z 
2025-07-17T11:07:26.6518030Z 
2025-07-17T11:07:26.6518121Z SIGABRT(6), PID: 1817868, Thread 1822146: 
2025-07-17T11:07:26.6518591Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6518795Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6518964Z frame #2: __poll + 0x4f (0x7f6a926ffbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6519179Z frame #3: <unknown function> + 0xd4540 (0x7f6956b12540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.6519381Z frame #4: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6519592Z frame #5: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6519597Z 
2025-07-17T11:07:26.6519689Z SIGABRT(6), PID: 1817873, Thread 1818206: 
2025-07-17T11:07:26.6520157Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6520366Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6520568Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6520769Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6521275Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6521479Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6521688Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6521694Z 
2025-07-17T11:07:26.6521784Z SIGABRT(6), PID: 1817867, Thread 1818037: 
2025-07-17T11:07:26.6522258Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6522467Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6522670Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6522867Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6523365Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6523570Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6523852Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6523857Z 
2025-07-17T11:07:26.6523949Z SIGABRT(6), PID: 1817868, Thread 1822153: 
2025-07-17T11:07:26.6524473Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6524738Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6524906Z frame #2: __poll + 0x4f (0x7f6a926ffbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6525182Z frame #3: <unknown function> + 0xd65a2 (0x7f6956b145a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.6525388Z frame #4: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6525601Z frame #5: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6525692Z SIGABRT(6), PID: 1817872, Thread 1818325: 
2025-07-17T11:07:26.6526159Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6526367Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6526574Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6526772Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6527277Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6527484Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6527693Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6527782Z SIGABRT(6), PID: 1817873, Thread 1818210: 
2025-07-17T11:07:26.6528250Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6528454Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6528658Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6528853Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6529351Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6529554Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6529761Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6529852Z SIGABRT(6), PID: 1817867, Thread 1818040: 
2025-07-17T11:07:26.6530325Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6530530Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6530736Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6530934Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6531430Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6531634Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6531919Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6531924Z 
2025-07-17T11:07:26.6531927Z 
2025-07-17T11:07:26.6531930Z 
2025-07-17T11:07:26.6531933Z 
2025-07-17T11:07:26.6532081Z SIGABRT(6), PID: 1817868, Thread 1822162: 
2025-07-17T11:07:26.6532556Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6532837Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6533101Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6533298Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6533514Z frame #4: <unknown function> + 0xd1e7f (0x7f6956b0fe7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.6533719Z frame #5: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6533932Z frame #6: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6533935Z 
2025-07-17T11:07:26.6534026Z SIGABRT(6), PID: 1817872, Thread 1818326: 
2025-07-17T11:07:26.6534497Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6534707Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6534913Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6535114Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6535609Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6535818Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6536089Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6536183Z SIGABRT(6), PID: 1817867, Thread 1818043: 
2025-07-17T11:07:26.6536649Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6536856Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6537057Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6537253Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6537748Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6537957Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6538164Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6538259Z SIGABRT(6), PID: 1817873, Thread 1818214: 
2025-07-17T11:07:26.6538728Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6538935Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6539136Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6539335Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6539831Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6540131Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6540411Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6540416Z 
2025-07-17T11:07:26.6540419Z 
2025-07-17T11:07:26.6540422Z 
2025-07-17T11:07:26.6540586Z SIGABRT(6), PID: 1817869, Thread 1817869: 
2025-07-17T11:07:26.6541060Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6541347Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6541580Z frame #2: <unknown function> + 0x7741f (0x7ff89e0f841f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6541811Z frame #3: <unknown function> + 0x76c8e (0x7ff89e0f7c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6542039Z frame #4: <unknown function> + 0x6ad69 (0x7ff89e0ebd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6542258Z frame #5: <unknown function> + 0x3f54db (0x7ff92ed504db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6542469Z frame #6: <unknown function> + 0x3e11d8 (0x7ff92ed3c1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6542684Z frame #7: <unknown function> + 0xb2f86 (0x7ff92ea0df86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6542891Z frame #8: <unknown function> + 0x8e501 (0x7ff92e9e9501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6543280Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7ff9597f07ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.6543670Z frame #10: <unknown function> + 0xd307d9 (0x7ff977d3a7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6543901Z frame #11: <unknown function> + 0x139902 (0x55f859e7a902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6544153Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55f859e71a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6544388Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6544630Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55f859e71a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6544859Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6545100Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6545324Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6545558Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6545782Z frame #19: <unknown function> + 0x1474b2 (0x55f859e884b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6545984Z frame #20: PyObject_Call + 0xbc (0x55f859e88e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6546213Z frame #21: <unknown function> + 0x1ff69d (0x55f859f4069d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6546435Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55f859e76023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6546680Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55f859e71cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6546903Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6547143Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55f859e6d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6547362Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6547596Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55f859e6d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6547893Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6548197Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55f859e71a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6548454Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55f859e754b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6548735Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55f859e86ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6548958Z frame #32: <unknown function> + 0x205949 (0x55f859f46949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6549246Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55f859e76023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6549483Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55f859e71cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6549711Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6549953Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55f859e6d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6550181Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6550415Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55f859e6d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6550642Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6550877Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55f859e6d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6551106Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6551344Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55f859e6e3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6551569Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6551954Z frame #44: <unknown function> + 0x7ab3f7 (0x7ff9777b53f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6552180Z frame #45: <unknown function> + 0x13bc88 (0x55f859e7cc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6552382Z frame #46: PyObject_Call + 0x20f (0x55f859e88f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6552629Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55f859e7298d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6552848Z frame #48: <unknown function> + 0x147608 (0x55f859e88608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6553089Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f859e6fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6553311Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6553549Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55f859e6d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6553775Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6554016Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55f859e6e3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6554237Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6554476Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55f859e6d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6554698Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6554941Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f859e6fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6555161Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6555403Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f859e6fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6555700Z frame #60: <unknown function> + 0x147608 (0x55f859e88608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6555999Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f859e6fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6556220Z frame #62: <unknown function> + 0x147608 (0x55f859e88608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6556224Z 
2025-07-17T11:07:26.6556377Z SIGABRT(6), PID: 1817870, Thread 1822237: 
2025-07-17T11:07:26.6556856Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6557139Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6557348Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6557555Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6557929Z frame #4: <unknown function> + 0x5a02023 (0x7f6af8693023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6558498Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6af86971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6559091Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6af868e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6559477Z frame #7: <unknown function> + 0x79d655 (0x7f6b09b7e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6559710Z frame #8: <unknown function> + 0xdc253 (0x7f6ac0bc9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6559921Z frame #9: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6560143Z frame #10: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6560149Z 
2025-07-17T11:07:26.6560241Z SIGABRT(6), PID: 1817868, Thread 1822171: 
2025-07-17T11:07:26.6560722Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6560928Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6561135Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6561338Z frame #3: <unknown function> + 0x9ca53 (0x7f6a92683a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6561589Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x559370a13bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6561815Z frame #5: <unknown function> + 0x1d314f (0x559370ad214f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6562037Z frame #6: <unknown function> + 0x1d2f6b (0x559370ad1f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6562254Z frame #7: <unknown function> + 0x6df36 (0x55937096cf36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6562495Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6562719Z frame #9: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6562960Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6563186Z frame #11: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6563423Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6563718Z frame #13: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6563954Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6564260Z frame #15: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6564502Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6564781Z frame #17: <unknown function> + 0x147734 (0x559370a46734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6565065Z frame #18: <unknown function> + 0x21ca82 (0x559370b1ba82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6565284Z frame #19: <unknown function> + 0x21c9e4 (0x559370b1b9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6565501Z frame #20: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6565710Z frame #21: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6565715Z 
2025-07-17T11:07:26.6565810Z SIGABRT(6), PID: 1817872, Thread 1818327: 
2025-07-17T11:07:26.6566279Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6566491Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6566697Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6566905Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6567407Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6567615Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6567829Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6567833Z 
2025-07-17T11:07:26.6567927Z SIGABRT(6), PID: 1817869, Thread 1817929: 
2025-07-17T11:07:26.6568394Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6568604Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6568806Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6569008Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6569504Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6569717Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6569926Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6570019Z SIGABRT(6), PID: 1817873, Thread 1818218: 
2025-07-17T11:07:26.6570485Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6570692Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6570894Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6571093Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6571589Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6571871Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6572140Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6572145Z 
2025-07-17T11:07:26.6572240Z SIGABRT(6), PID: 1817867, Thread 1818046: 
2025-07-17T11:07:26.6572767Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6572980Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6573276Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6573476Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6573970Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6574181Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6574392Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6574485Z SIGABRT(6), PID: 1817868, Thread 1822184: 
2025-07-17T11:07:26.6574957Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6575165Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6575327Z frame #2: read + 0x4c (0x7f6a926fb81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6575508Z frame #3: _Py_read + 0x4c (0x559370ad9a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6575730Z frame #4: <unknown function> + 0x208c3b (0x559370b07c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6576022Z frame #5: <unknown function> + 0x14648d (0x559370a4548d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6576269Z frame #6: PyObject_VectorcallMethod + 0x85 (0x559370a54215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6576490Z frame #7: <unknown function> + 0x208ae7 (0x559370b07ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6576707Z frame #8: <unknown function> + 0x208472 (0x559370b07472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6576932Z frame #9: <unknown function> + 0x1da6ee (0x559370ad96ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6577157Z frame #10: <unknown function> + 0x1483bc (0x559370a473bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6577403Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6577631Z frame #12: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6577870Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x559370a2b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6578087Z frame #14: <unknown function> + 0x147734 (0x559370a46734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6578337Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x559370a2dc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6578560Z frame #16: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6578797Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6579021Z frame #18: _PyFunction_Vectorcall + 0x6c (0x559370a3b0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6579257Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x559370a2b7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6579472Z frame #20: <unknown function> + 0x147734 (0x559370a46734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6579783Z frame #21: <unknown function> + 0x21ca82 (0x559370b1ba82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6580074Z frame #22: <unknown function> + 0x21c9e4 (0x559370b1b9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6580292Z frame #23: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6580572Z frame #24: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6580672Z SIGABRT(6), PID: 1817872, Thread 1818328: 
2025-07-17T11:07:26.6586096Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6586369Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6586589Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6586811Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6587337Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6587555Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6587772Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6587777Z 
2025-07-17T11:07:26.6587787Z 
2025-07-17T11:07:26.6587790Z 
2025-07-17T11:07:26.6587793Z 
2025-07-17T11:07:26.6587890Z SIGABRT(6), PID: 1817869, Thread 1817933: 
2025-07-17T11:07:26.6588398Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6588616Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6588825Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6589028Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6589539Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6589756Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6589981Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6589985Z 
2025-07-17T11:07:26.6590079Z SIGABRT(6), PID: 1817872, Thread 1818329: 
2025-07-17T11:07:26.6590568Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6590782Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6590988Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6591192Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6591703Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6591913Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6592126Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6592130Z 
2025-07-17T11:07:26.6592219Z SIGABRT(6), PID: 1817867, Thread 1818049: 
2025-07-17T11:07:26.6592693Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6593025Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6593299Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6593501Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6594061Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6594325Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6594539Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6594544Z 
2025-07-17T11:07:26.6594631Z SIGABRT(6), PID: 1817873, Thread 1818222: 
2025-07-17T11:07:26.6595104Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6595310Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6595511Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6595708Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6596205Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6596407Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6596613Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6596617Z 
2025-07-17T11:07:26.6596705Z SIGABRT(6), PID: 1817872, Thread 1818330: 
2025-07-17T11:07:26.6597169Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6597377Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6597582Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6597780Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6598269Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6598475Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6598680Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6598687Z 
2025-07-17T11:07:26.6598777Z SIGABRT(6), PID: 1817869, Thread 1817937: 
2025-07-17T11:07:26.6599238Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6599442Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6599644Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6599838Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6600329Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6600537Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6600844Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6600848Z 
2025-07-17T11:07:26.6600939Z SIGABRT(6), PID: 1817873, Thread 1818226: 
2025-07-17T11:07:26.6601459Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6601670Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6601929Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6602182Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6602673Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6602876Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6603083Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6603088Z 
2025-07-17T11:07:26.6603179Z SIGABRT(6), PID: 1817867, Thread 1818052: 
2025-07-17T11:07:26.6603640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6603848Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6604049Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6604243Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6604731Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6604936Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6605143Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6605235Z SIGABRT(6), PID: 1817872, Thread 1818331: 
2025-07-17T11:07:26.6605697Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6605906Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6606108Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6606304Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6606791Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6607002Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6607207Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6607214Z 
2025-07-17T11:07:26.6607218Z 
2025-07-17T11:07:26.6607304Z SIGABRT(6), PID: 1817873, Thread 1818230: 
2025-07-17T11:07:26.6607769Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6607973Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6608176Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6608368Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6608857Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6609193Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6609401Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6609405Z 
2025-07-17T11:07:26.6609493Z SIGABRT(6), PID: 1817869, Thread 1817941: 
2025-07-17T11:07:26.6610021Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6610283Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6610491Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6610686Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6611182Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6611387Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6611593Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6611597Z 
2025-07-17T11:07:26.6611686Z SIGABRT(6), PID: 1817872, Thread 1818332: 
2025-07-17T11:07:26.6612152Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6612364Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6612569Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6612766Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6613267Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6613471Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6613685Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6613689Z 
2025-07-17T11:07:26.6613776Z SIGABRT(6), PID: 1817867, Thread 1818055: 
2025-07-17T11:07:26.6614247Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6614452Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6614654Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6614848Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6615349Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6615553Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6615760Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6615765Z 
2025-07-17T11:07:26.6615851Z SIGABRT(6), PID: 1817873, Thread 1818235: 
2025-07-17T11:07:26.6616426Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6616631Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6616930Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6617197Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6617695Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6617968Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6618248Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6618253Z 
2025-07-17T11:07:26.6618342Z SIGABRT(6), PID: 1817869, Thread 1817945: 
2025-07-17T11:07:26.6618805Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6619012Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6619212Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6619411Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6619904Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6620109Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6620315Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6620406Z SIGABRT(6), PID: 1817870, Thread 1822238: 
2025-07-17T11:07:26.6620866Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6621072Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6621274Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6621468Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6621841Z frame #4: <unknown function> + 0x5a02023 (0x7f6af8693023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6622409Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6af86971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6623003Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6af868e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6623388Z frame #7: <unknown function> + 0x79d655 (0x7f6b09b7e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6623623Z frame #8: <unknown function> + 0xdc253 (0x7f6ac0bc9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6623832Z frame #9: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6624048Z frame #10: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6624052Z 
2025-07-17T11:07:26.6624056Z 
2025-07-17T11:07:26.6624147Z SIGABRT(6), PID: 1817872, Thread 1818333: 
2025-07-17T11:07:26.6624616Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6624824Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6625103Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6625303Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6625860Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6626129Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6626342Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6626425Z 
2025-07-17T11:07:26.6626517Z SIGABRT(6), PID: 1817867, Thread 1818058: 
2025-07-17T11:07:26.6626981Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6627188Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6627390Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6627586Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6628082Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6628288Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6628496Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6628500Z 
2025-07-17T11:07:26.6628587Z SIGABRT(6), PID: 1817873, Thread 1818239: 
2025-07-17T11:07:26.6629051Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6629255Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6629461Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6629654Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6630153Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6630354Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6630561Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6630565Z 
2025-07-17T11:07:26.6630651Z SIGABRT(6), PID: 1817869, Thread 1817949: 
2025-07-17T11:07:26.6631119Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6631324Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6631531Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6631724Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6632218Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6632423Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6632629Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6632634Z 
2025-07-17T11:07:26.6632722Z SIGABRT(6), PID: 1817872, Thread 1818334: 
2025-07-17T11:07:26.6633259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6633525Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6633734Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6633988Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6634488Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6634759Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6634968Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6634972Z 
2025-07-17T11:07:26.6635062Z SIGABRT(6), PID: 1817868, Thread 1822228: 
2025-07-17T11:07:26.6635529Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6635735Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6635935Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6636129Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6636504Z frame #4: <unknown function> + 0x5a02023 (0x7f69d6aa8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6637062Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f69d6aac1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6637664Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f69d6aa31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6638046Z frame #7: <unknown function> + 0x79d655 (0x7f69e7f8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6638277Z frame #8: <unknown function> + 0xdc253 (0x7f699efde253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6638483Z frame #9: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6638698Z frame #10: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6638791Z SIGABRT(6), PID: 1817867, Thread 1818060: 
2025-07-17T11:07:26.6639264Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6639472Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6639672Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6639871Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6640362Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6640566Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6640775Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6640779Z 
2025-07-17T11:07:26.6640782Z 
2025-07-17T11:07:26.6640871Z SIGABRT(6), PID: 1817873, Thread 1818243: 
2025-07-17T11:07:26.6641331Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6641615Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6641873Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6642070Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6642625Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6643012Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6643217Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6643221Z 
2025-07-17T11:07:26.6643309Z SIGABRT(6), PID: 1817869, Thread 1817953: 
2025-07-17T11:07:26.6643771Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6643978Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6644179Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6644375Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6644870Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6645080Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6645286Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6645292Z 
2025-07-17T11:07:26.6645381Z SIGABRT(6), PID: 1817872, Thread 1818335: 
2025-07-17T11:07:26.6645845Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6646050Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6646251Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6646449Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6646946Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6647154Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6647359Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6647450Z SIGABRT(6), PID: 1817873, Thread 1818247: 
2025-07-17T11:07:26.6647918Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6648123Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6648321Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6648514Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6649006Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6649210Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6649495Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6649499Z 
2025-07-17T11:07:26.6649502Z 
2025-07-17T11:07:26.6649588Z SIGABRT(6), PID: 1817867, Thread 1818063: 
2025-07-17T11:07:26.6650113Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6650377Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6650582Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6650837Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6651329Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6651534Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6651741Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6651745Z 
2025-07-17T11:07:26.6651834Z SIGABRT(6), PID: 1817873, Thread 1818251: 
2025-07-17T11:07:26.6652299Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6652502Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6652706Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6652896Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6653390Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6653592Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6653799Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6653803Z 
2025-07-17T11:07:26.6653888Z SIGABRT(6), PID: 1817869, Thread 1817957: 
2025-07-17T11:07:26.6654355Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6654556Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6654759Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6654952Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6655445Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6655650Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6655860Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6656020Z SIGABRT(6), PID: 1817867, Thread 1818066: 
2025-07-17T11:07:26.6656489Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6656696Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6656898Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6657091Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6657579Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6657948Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6658163Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6658250Z SIGABRT(6), PID: 1817872, Thread 1818336: 
2025-07-17T11:07:26.6658832Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6659114Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6659319Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6659515Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6660013Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6660221Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6660428Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6660516Z SIGABRT(6), PID: 1817873, Thread 1818255: 
2025-07-17T11:07:26.6660981Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6661185Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6661382Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6661575Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6662068Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6662273Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6662476Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6662482Z 
2025-07-17T11:07:26.6662486Z 
2025-07-17T11:07:26.6662489Z 
2025-07-17T11:07:26.6662492Z 
2025-07-17T11:07:26.6662582Z SIGABRT(6), PID: 1817870, Thread 1822240: 
2025-07-17T11:07:26.6663043Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6663248Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6663454Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6663650Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6664022Z frame #4: <unknown function> + 0x5a02023 (0x7f6af8693023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6664583Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6af86971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6665174Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6af868e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6665554Z frame #7: <unknown function> + 0x79d655 (0x7f6b09b7e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6665866Z frame #8: <unknown function> + 0xdc253 (0x7f6ac0bc9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6666140Z frame #9: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6666355Z frame #10: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6666360Z 
2025-07-17T11:07:26.6666449Z SIGABRT(6), PID: 1817873, Thread 1818259: 
2025-07-17T11:07:26.6666977Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6667241Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6667438Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6667632Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6668128Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6668335Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6668540Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6668547Z 
2025-07-17T11:07:26.6668636Z SIGABRT(6), PID: 1817867, Thread 1818069: 
2025-07-17T11:07:26.6669103Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6669310Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6669512Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6669703Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6670202Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6670403Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6670611Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6670615Z 
2025-07-17T11:07:26.6670700Z SIGABRT(6), PID: 1817872, Thread 1818337: 
2025-07-17T11:07:26.6671170Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6671373Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6671576Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6671773Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6672267Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6672472Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6672680Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6672686Z 
2025-07-17T11:07:26.6672773Z SIGABRT(6), PID: 1817869, Thread 1817961: 
2025-07-17T11:07:26.6673236Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6673437Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6673717Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6673912Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6676382Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6676683Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6676912Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6676918Z 
2025-07-17T11:07:26.6677015Z SIGABRT(6), PID: 1817873, Thread 1818262: 
2025-07-17T11:07:26.6677508Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6677726Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6677932Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6678133Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6678655Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6678865Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6679071Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6679076Z 
2025-07-17T11:07:26.6679170Z SIGABRT(6), PID: 1817867, Thread 1818072: 
2025-07-17T11:07:26.6679644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6679855Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6680058Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6680264Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6680759Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6680964Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6681172Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6681176Z 
2025-07-17T11:07:26.6681264Z SIGABRT(6), PID: 1817872, Thread 1818338: 
2025-07-17T11:07:26.6681730Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6681938Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6682147Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6682346Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6682844Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6683051Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6683261Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6683265Z 
2025-07-17T11:07:26.6683353Z SIGABRT(6), PID: 1817869, Thread 1817965: 
2025-07-17T11:07:26.6683896Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6684163Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6684494Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6684759Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6685275Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6685489Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6685702Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6685709Z 
2025-07-17T11:07:26.6685797Z SIGABRT(6), PID: 1817873, Thread 1818266: 
2025-07-17T11:07:26.6686272Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6686480Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6686682Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6686880Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6687377Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6687582Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6687788Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6687794Z 
2025-07-17T11:07:26.6687882Z SIGABRT(6), PID: 1817867, Thread 1818075: 
2025-07-17T11:07:26.6688351Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6688559Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6688759Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6688954Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6689447Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6689654Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6689861Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6689951Z SIGABRT(6), PID: 1817868, Thread 1822229: 
2025-07-17T11:07:26.6690417Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6690622Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6690823Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6691019Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6691394Z frame #4: <unknown function> + 0x5a02023 (0x7f69d6aa8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6691957Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f69d6aac1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6692708Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f69d6aa31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6693220Z frame #7: <unknown function> + 0x79d655 (0x7f69e7f8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6693463Z frame #8: <unknown function> + 0xdc253 (0x7f699efde253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6695151Z frame #9: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6695381Z frame #10: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6695386Z 
2025-07-17T11:07:26.6695390Z 
2025-07-17T11:07:26.6695492Z SIGABRT(6), PID: 1817872, Thread 1818339: 
2025-07-17T11:07:26.6696068Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6696293Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6696501Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6696710Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6697227Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6697459Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6697678Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6697682Z 
2025-07-17T11:07:26.6697774Z SIGABRT(6), PID: 1817873, Thread 1818270: 
2025-07-17T11:07:26.6698263Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6698472Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6698678Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6698878Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6699383Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6699590Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6699800Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6699805Z 
2025-07-17T11:07:26.6699896Z SIGABRT(6), PID: 1817869, Thread 1817969: 
2025-07-17T11:07:26.6700376Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6700587Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6700791Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6700993Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6701497Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6701711Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6701920Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6701925Z 
2025-07-17T11:07:26.6702015Z SIGABRT(6), PID: 1817867, Thread 1818078: 
2025-07-17T11:07:26.6702686Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6702976Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6703186Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6703490Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6703986Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6704194Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6704398Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6704405Z 
2025-07-17T11:07:26.6704498Z SIGABRT(6), PID: 1817872, Thread 1818340: 
2025-07-17T11:07:26.6704970Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6705184Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6705386Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6705589Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6706086Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6706296Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6706505Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6706511Z 
2025-07-17T11:07:26.6706604Z SIGABRT(6), PID: 1817869, Thread 1817972: 
2025-07-17T11:07:26.6707073Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6707282Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6707484Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6707681Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6708176Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6708387Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6708594Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6708685Z SIGABRT(6), PID: 1817873, Thread 1818275: 
2025-07-17T11:07:26.6709146Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6709353Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6709556Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6709756Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6710246Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6710524Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6710794Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6710798Z 
2025-07-17T11:07:26.6710948Z SIGABRT(6), PID: 1817867, Thread 1818081: 
2025-07-17T11:07:26.6711417Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6711700Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6711903Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6712105Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6712600Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6712811Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6713018Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6713109Z SIGABRT(6), PID: 1817872, Thread 1818341: 
2025-07-17T11:07:26.6713579Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6713787Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6713993Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6714189Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6714684Z frame #4: <unknown function> + 0x3475ab (0x7fdc92a265ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6714893Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6715104Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6715109Z 
2025-07-17T11:07:26.6715112Z 
2025-07-17T11:07:26.6715116Z 
2025-07-17T11:07:26.6715203Z SIGABRT(6), PID: 1817872, Thread 1818400: 
2025-07-17T11:07:26.6715675Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6716129Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fdba428508a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6716339Z frame #2: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6716533Z frame #3: pthread_kill + 0x12c (0x7fdc95b909fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6716704Z frame #4: raise + 0x16 (0x7fdc95b3c476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6716863Z frame #5: abort + 0xd3 (0x7fdc95b227f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6717107Z frame #6: <unknown function> + 0x972cc (0x7fdb11e0e2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6717338Z frame #7: <unknown function> + 0x95f47 (0x7fdb11e0cf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6717568Z frame #8: <unknown function> + 0x34b71 (0x7fdb11dabb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6717778Z frame #9: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6717996Z frame #10: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6718000Z 
2025-07-17T11:07:26.6718092Z SIGABRT(6), PID: 1817869, Thread 1817975: 
2025-07-17T11:07:26.6718655Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6718992Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6719201Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6719402Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6719991Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6720199Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6720407Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6720412Z 
2025-07-17T11:07:26.6720506Z SIGABRT(6), PID: 1817867, Thread 1818084: 
2025-07-17T11:07:26.6720979Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6721185Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6721390Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6721590Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6722089Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6722295Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6722506Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6722512Z 
2025-07-17T11:07:26.6722600Z SIGABRT(6), PID: 1817873, Thread 1818279: 
2025-07-17T11:07:26.6723070Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6723276Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6723479Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6723675Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6724170Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6724381Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6724587Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6724593Z 
2025-07-17T11:07:26.6724686Z SIGABRT(6), PID: 1817870, Thread 1822242: 
2025-07-17T11:07:26.6725156Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6725362Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6725566Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6725764Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6726144Z frame #4: <unknown function> + 0x5a02023 (0x7f6af8693023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6726788Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6af86971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6727501Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6af868e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6727895Z frame #7: <unknown function> + 0x79d655 (0x7f6b09b7e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6728194Z frame #8: <unknown function> + 0xdc253 (0x7f6ac0bc9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6728407Z frame #9: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6728626Z frame #10: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6728631Z 
2025-07-17T11:07:26.6728728Z SIGABRT(6), PID: 1817872, Thread 1821913: 
2025-07-17T11:07:26.6729197Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6729411Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6729580Z frame #2: __poll + 0x4f (0x7fdc95c12bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6729990Z frame #3: <unknown function> + 0x489f (0x7fdc9571e89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.6730223Z frame #4: <unknown function> + 0x1483bc (0x5626a8da13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6730470Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6730696Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6730936Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6731157Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6731407Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5626a8d89a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6731637Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6731892Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5626a8d87c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6732118Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6732358Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6732579Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6732822Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6733040Z frame #16: <unknown function> + 0x147734 (0x5626a8da0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6733264Z frame #17: <unknown function> + 0x21ca82 (0x5626a8e75a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6733479Z frame #18: <unknown function> + 0x21c9e4 (0x5626a8e759e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6733697Z frame #19: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6733909Z frame #20: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6733913Z 
2025-07-17T11:07:26.6734006Z SIGABRT(6), PID: 1817867, Thread 1818087: 
2025-07-17T11:07:26.6734484Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6734770Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6735042Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6735304Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6735807Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6736171Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6736387Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6736483Z SIGABRT(6), PID: 1817869, Thread 1817979: 
2025-07-17T11:07:26.6736965Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6737179Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6737387Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6737593Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6738099Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6738311Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6738523Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6738529Z 
2025-07-17T11:07:26.6738533Z 
2025-07-17T11:07:26.6738620Z SIGABRT(6), PID: 1817873, Thread 1818283: 
2025-07-17T11:07:26.6739098Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6739312Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6739518Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6739715Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6740229Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6740435Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6740642Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6740646Z 
2025-07-17T11:07:26.6740734Z SIGABRT(6), PID: 1817867, Thread 1818090: 
2025-07-17T11:07:26.6741205Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6741414Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6741620Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6741819Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6742319Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6742526Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6742737Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6742741Z 
2025-07-17T11:07:26.6742942Z SIGABRT(6), PID: 1817873, Thread 1818286: 
2025-07-17T11:07:26.6743489Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6743768Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6743977Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6744255Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6744754Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6744959Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6745167Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6745173Z 
2025-07-17T11:07:26.6745261Z SIGABRT(6), PID: 1817869, Thread 1817982: 
2025-07-17T11:07:26.6745733Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6745937Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6746145Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6746343Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6746840Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6747045Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6747254Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6747260Z 
2025-07-17T11:07:26.6747347Z SIGABRT(6), PID: 1817868, Thread 1822230: 
2025-07-17T11:07:26.6747819Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6748020Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6748223Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6748418Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6748802Z frame #4: <unknown function> + 0x5a02023 (0x7f69d6aa8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6749373Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f69d6aac1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6749974Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f69d6aa31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6750359Z frame #7: <unknown function> + 0x79d655 (0x7f69e7f8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6750594Z frame #8: <unknown function> + 0xdc253 (0x7f699efde253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6750803Z frame #9: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6751019Z frame #10: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6751023Z 
2025-07-17T11:07:26.6751115Z SIGABRT(6), PID: 1817873, Thread 1818290: 
2025-07-17T11:07:26.6751722Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6752018Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6752227Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6752487Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6752984Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6753189Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6753396Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6753489Z SIGABRT(6), PID: 1817867, Thread 1818093: 
2025-07-17T11:07:26.6753961Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6754169Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6754374Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6754589Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6755085Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6755293Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6755501Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6755507Z 
2025-07-17T11:07:26.6755510Z 
2025-07-17T11:07:26.6755601Z SIGABRT(6), PID: 1817869, Thread 1817986: 
2025-07-17T11:07:26.6756072Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6756278Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6756483Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6756679Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6757172Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6757383Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6757592Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6757596Z 
2025-07-17T11:07:26.6757686Z SIGABRT(6), PID: 1817873, Thread 1818294: 
2025-07-17T11:07:26.6758157Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6758362Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6758563Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6758758Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6759252Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6759528Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6759799Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6759803Z 
2025-07-17T11:07:26.6759950Z SIGABRT(6), PID: 1817866, Thread 1817866: 
2025-07-17T11:07:26.6760431Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6760705Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6760942Z frame #2: <unknown function> + 0x7744b (0x7fec9c74f44b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6761176Z frame #3: <unknown function> + 0x76c8e (0x7fec9c74ec8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6761404Z frame #4: <unknown function> + 0x6ad69 (0x7fec9c742d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6761621Z frame #5: <unknown function> + 0x3f54db (0x7fed2d3a74db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6761837Z frame #6: <unknown function> + 0x3e11d8 (0x7fed2d3931d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6762047Z frame #7: <unknown function> + 0xb2f86 (0x7fed2d064f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6762256Z frame #8: <unknown function> + 0x8e501 (0x7fed2d040501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6762643Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fed57e477ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.6763035Z frame #10: <unknown function> + 0xd307d9 (0x7fed7639a7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6763260Z frame #11: <unknown function> + 0x139902 (0x5622549bb902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6763516Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5622549b2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6763746Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6763995Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5622549b2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6764220Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6764459Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6764682Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6764921Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6765139Z frame #19: <unknown function> + 0x1474b2 (0x5622549c94b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6765342Z frame #20: PyObject_Call + 0xbc (0x5622549c9e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6765566Z frame #21: <unknown function> + 0x1ff69d (0x562254a8169d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6765793Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5622549b7023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6766033Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5622549b2cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6766259Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6766493Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5622549ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6766717Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6766951Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5622549ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6767177Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6767488Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5622549b2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6767810Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5622549b64b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6768092Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5622549c7ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6768314Z frame #32: <unknown function> + 0x205949 (0x562254a87949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6768598Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5622549b7023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6768837Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5622549b2cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6769058Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6769304Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5622549ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6769529Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6769768Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5622549ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6769991Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6770228Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5622549ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6770451Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6770692Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5622549af3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6770914Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6771308Z frame #44: <unknown function> + 0x7ab3f7 (0x7fed75e153f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6771529Z frame #45: <unknown function> + 0x13bc88 (0x5622549bdc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6771732Z frame #46: PyObject_Call + 0x20f (0x5622549c9f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6771976Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5622549b398d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6772193Z frame #48: <unknown function> + 0x147608 (0x5622549c9608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6772431Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5622549b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6772656Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6772891Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5622549ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6773122Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6773361Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5622549af3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6773588Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6773826Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5622549ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6774056Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6774297Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5622549b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6774521Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6774760Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5622549b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6774978Z frame #60: <unknown function> + 0x147608 (0x5622549c9608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6775286Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5622549b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6775677Z frame #62: <unknown function> + 0x147608 (0x5622549c9608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6775828Z SIGABRT(6), PID: 1817867, Thread 1818097: 
2025-07-17T11:07:26.6776399Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6776697Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6776907Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6777109Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6777613Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6777822Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6778032Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6778037Z 
2025-07-17T11:07:26.6778040Z 
2025-07-17T11:07:26.6778133Z SIGABRT(6), PID: 1817872, Thread 1821914: 
2025-07-17T11:07:26.6778608Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6778818Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6779021Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6779248Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fdc95b8e2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6779694Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fdbcffcd048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6780122Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fdbcffcfbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6780355Z frame #6: <unknown function> + 0xdc253 (0x7fdba24f1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6780568Z frame #7: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6780777Z frame #8: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6780781Z 
2025-07-17T11:07:26.6780873Z SIGABRT(6), PID: 1817869, Thread 1817988: 
2025-07-17T11:07:26.6781346Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6781555Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6781758Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6781962Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6782464Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6782678Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6782884Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6782888Z 
2025-07-17T11:07:26.6782978Z SIGABRT(6), PID: 1817867, Thread 1818101: 
2025-07-17T11:07:26.6783529Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6783837Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6784117Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6784335Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6784849Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6785121Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6785331Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6785424Z SIGABRT(6), PID: 1817873, Thread 1818298: 
2025-07-17T11:07:26.6785892Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6786101Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6786304Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6786502Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6786997Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6787204Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6787420Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6787427Z 
2025-07-17T11:07:26.6787430Z 
2025-07-17T11:07:26.6787521Z SIGABRT(6), PID: 1817869, Thread 1817990: 
2025-07-17T11:07:26.6787998Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6788203Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6788407Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6788605Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6789105Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6789311Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6789531Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6789535Z 
2025-07-17T11:07:26.6789626Z SIGABRT(6), PID: 1817867, Thread 1818106: 
2025-07-17T11:07:26.6790093Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6790299Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6790504Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6790702Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6791198Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6791401Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6791683Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6791835Z SIGABRT(6), PID: 1817866, Thread 1817874: 
2025-07-17T11:07:26.6792370Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6792579Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6792849Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6793049Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6793547Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6793757Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6793968Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6793974Z 
2025-07-17T11:07:26.6793977Z 
2025-07-17T11:07:26.6794064Z SIGABRT(6), PID: 1817873, Thread 1818300: 
2025-07-17T11:07:26.6794539Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6794742Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6794945Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6795142Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6795637Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6795843Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6796051Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6796055Z 
2025-07-17T11:07:26.6796146Z SIGABRT(6), PID: 1817872, Thread 1821915: 
2025-07-17T11:07:26.6796614Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6796823Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6797026Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6797249Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fdc95b8e2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6797724Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fdbcffc7741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6797959Z frame #5: <unknown function> + 0xdc253 (0x7fdba24f1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6798167Z frame #6: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6798378Z frame #7: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6798382Z 
2025-07-17T11:07:26.6798471Z SIGABRT(6), PID: 1817869, Thread 1817994: 
2025-07-17T11:07:26.6798941Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6799145Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6799348Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6799617Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6800177Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6800442Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6800657Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6800719Z 
2025-07-17T11:07:26.6800811Z SIGABRT(6), PID: 1817872, Thread 1821916: 
2025-07-17T11:07:26.6801289Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6801494Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6801665Z frame #2: ioctl + 0x3f (0x7fdc95c1494f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6801907Z frame #3: <unknown function> + 0x13eef0 (0x7fdb11eb5ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6802140Z frame #4: <unknown function> + 0x1378c4 (0x7fdb11eae8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6802369Z frame #5: <unknown function> + 0x95b2b (0x7fdb11e0cb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6802597Z frame #6: <unknown function> + 0x34b71 (0x7fdb11dabb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6802805Z frame #7: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6803014Z frame #8: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6803018Z 
2025-07-17T11:07:26.6803106Z SIGABRT(6), PID: 1817867, Thread 1818110: 
2025-07-17T11:07:26.6803579Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6803786Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6803991Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6804189Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6804687Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6804893Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6805101Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6805188Z SIGABRT(6), PID: 1817870, Thread 1822244: 
2025-07-17T11:07:26.6805659Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6805863Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6806068Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6806263Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6806640Z frame #4: <unknown function> + 0x5a02023 (0x7f6af8693023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6807205Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6af86971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6807872Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6af868e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6808315Z frame #7: <unknown function> + 0x79d655 (0x7f6b09b7e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6808606Z frame #8: <unknown function> + 0xdc253 (0x7f6ac0bc9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6808821Z frame #9: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6809105Z frame #10: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6809195Z SIGABRT(6), PID: 1817873, Thread 1818304: 
2025-07-17T11:07:26.6809670Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6809874Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6810080Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6810278Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6810777Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6810983Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6811190Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6811279Z SIGABRT(6), PID: 1817871, Thread 1817871: 
2025-07-17T11:07:26.6811755Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6811963Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6812191Z frame #2: <unknown function> + 0x77435 (0x7fa246d27435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6812422Z frame #3: <unknown function> + 0x76c8e (0x7fa246d26c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6812647Z frame #4: <unknown function> + 0x6ad69 (0x7fa246d1ad69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6812863Z frame #5: <unknown function> + 0x3f54db (0x7fa2d797f4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6813073Z frame #6: <unknown function> + 0x3e11d8 (0x7fa2d796b1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6813284Z frame #7: <unknown function> + 0xb2f86 (0x7fa2d763cf86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6813488Z frame #8: <unknown function> + 0x8e501 (0x7fa2d7618501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.6813868Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fa30241f7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.6814249Z frame #10: <unknown function> + 0xd307d9 (0x7fa3209727d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6814476Z frame #11: <unknown function> + 0x139902 (0x556e5a966902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6814725Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x556e5a95da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6814956Z frame #13: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6815199Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x556e5a95da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6815428Z frame #15: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6815662Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6815887Z frame #17: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6816372Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6816597Z frame #19: <unknown function> + 0x1474b2 (0x556e5a9744b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6816872Z frame #20: PyObject_Call + 0xbc (0x556e5a974e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6817103Z frame #21: <unknown function> + 0x1ff69d (0x556e5aa2c69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6817418Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x556e5a962023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6817661Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x556e5a95dcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6817880Z frame #24: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6818116Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x556e5a9593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6818339Z frame #26: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6818575Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x556e5a9593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6818798Z frame #28: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6819041Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x556e5a95da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6819298Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x556e5a9614b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6819518Z frame #31: _PyObject_Call_Prepend + 0x69 (0x556e5a972ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6819737Z frame #32: <unknown function> + 0x205949 (0x556e5aa32949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6819958Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x556e5a962023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6820198Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x556e5a95dcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6820423Z frame #35: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6820660Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x556e5a9593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6820886Z frame #37: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6821124Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x556e5a9593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6821348Z frame #39: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6821583Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x556e5a9593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6821810Z frame #41: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6822053Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x556e5a95a3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6822278Z frame #43: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6822665Z frame #44: <unknown function> + 0x7ab3f7 (0x7fa3203ed3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6822886Z frame #45: <unknown function> + 0x13bc88 (0x556e5a968c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6823091Z frame #46: PyObject_Call + 0x20f (0x556e5a974f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6823335Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x556e5a95e98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6823550Z frame #48: <unknown function> + 0x147608 (0x556e5a974608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6823792Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x556e5a95bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6824088Z frame #50: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6824407Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x556e5a9593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6824688Z frame #52: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6824931Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x556e5a95a3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6825220Z frame #54: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6825456Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x556e5a9593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6825677Z frame #56: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6825919Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x556e5a95bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6826141Z frame #58: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6826384Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x556e5a95bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6826598Z frame #60: <unknown function> + 0x147608 (0x556e5a974608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6826836Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x556e5a95bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6827057Z frame #62: <unknown function> + 0x147608 (0x556e5a974608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6827061Z 
2025-07-17T11:07:26.6827064Z 
2025-07-17T11:07:26.6827067Z 
2025-07-17T11:07:26.6827163Z SIGABRT(6), PID: 1817866, Thread 1817875: 
2025-07-17T11:07:26.6827643Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6827860Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6828069Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6828271Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6828770Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6828984Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6829192Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6829196Z 
2025-07-17T11:07:26.6829199Z 
2025-07-17T11:07:26.6829292Z SIGABRT(6), PID: 1817869, Thread 1817997: 
2025-07-17T11:07:26.6829773Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6829982Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6830186Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6830386Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6830886Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6831096Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6831306Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6831310Z 
2025-07-17T11:07:26.6831403Z SIGABRT(6), PID: 1817873, Thread 1818307: 
2025-07-17T11:07:26.6831941Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6832213Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6832485Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6832685Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6833245Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6833452Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6833660Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6833667Z 
2025-07-17T11:07:26.6833758Z SIGABRT(6), PID: 1817872, Thread 1822144: 
2025-07-17T11:07:26.6834235Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6834443Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6834612Z frame #2: __poll + 0x4f (0x7fdc95c12bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6834827Z frame #3: <unknown function> + 0xd4540 (0x7fdb5a025540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.6835038Z frame #4: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6835244Z frame #5: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6835247Z 
2025-07-17T11:07:26.6835338Z SIGABRT(6), PID: 1817867, Thread 1818114: 
2025-07-17T11:07:26.6835805Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6836013Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6836220Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6836420Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6836914Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6837124Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6837331Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6837335Z 
2025-07-17T11:07:26.6837425Z SIGABRT(6), PID: 1817871, Thread 1818105: 
2025-07-17T11:07:26.6837893Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6838103Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6838308Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6838506Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6838998Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6839206Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6839420Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6839424Z 
2025-07-17T11:07:26.6839587Z SIGABRT(6), PID: 1817866, Thread 1817876: 
2025-07-17T11:07:26.6840121Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6840389Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6840598Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6840885Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6841378Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6841587Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6841792Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6841798Z 
2025-07-17T11:07:26.6841891Z SIGABRT(6), PID: 1817868, Thread 1822231: 
2025-07-17T11:07:26.6842359Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6842565Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6842768Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6842968Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6843343Z frame #4: <unknown function> + 0x5a02023 (0x7f69d6aa8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6843912Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f69d6aac1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6844505Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f69d6aa31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6844885Z frame #7: <unknown function> + 0x79d655 (0x7f69e7f8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6845123Z frame #8: <unknown function> + 0xdc253 (0x7f699efde253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6845330Z frame #9: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6845543Z frame #10: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6845548Z 
2025-07-17T11:07:26.6845635Z SIGABRT(6), PID: 1817872, Thread 1822152: 
2025-07-17T11:07:26.6846112Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6846318Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6846490Z frame #2: __poll + 0x4f (0x7fdc95c12bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6846705Z frame #3: <unknown function> + 0xd65a2 (0x7fdb5a0275a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.6846915Z frame #4: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6847121Z frame #5: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6847124Z 
2025-07-17T11:07:26.6847216Z SIGABRT(6), PID: 1817869, Thread 1818000: 
2025-07-17T11:07:26.6847684Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6847964Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6848237Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6848499Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6849001Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6849274Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6849484Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6849488Z 
2025-07-17T11:07:26.6849580Z SIGABRT(6), PID: 1817873, Thread 1818310: 
2025-07-17T11:07:26.6850055Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6850264Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6850463Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6850663Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6851161Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6851370Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6851575Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6851579Z 
2025-07-17T11:07:26.6851669Z SIGABRT(6), PID: 1817867, Thread 1818117: 
2025-07-17T11:07:26.6852135Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6852343Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6852546Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6852744Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6853241Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6853448Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6853654Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6853658Z 
2025-07-17T11:07:26.6853748Z SIGABRT(6), PID: 1817872, Thread 1822156: 
2025-07-17T11:07:26.6854220Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6854429Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6854633Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6854833Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6855049Z frame #4: <unknown function> + 0xd1e7f (0x7fdb5a022e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.6855257Z frame #5: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6855465Z frame #6: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6855471Z 
2025-07-17T11:07:26.6855558Z SIGABRT(6), PID: 1817871, Thread 1818108: 
2025-07-17T11:07:26.6856188Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6856541Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6856755Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6856954Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6857525Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6857735Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6857943Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6857947Z 
2025-07-17T11:07:26.6858037Z SIGABRT(6), PID: 1817869, Thread 1818004: 
2025-07-17T11:07:26.6858511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6858718Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6858922Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6859120Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6859626Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6859833Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6860044Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6860050Z 
2025-07-17T11:07:26.6860137Z SIGABRT(6), PID: 1817866, Thread 1817877: 
2025-07-17T11:07:26.6860611Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6860813Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6861020Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6861216Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6861713Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6861916Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6862128Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6862134Z 
2025-07-17T11:07:26.6862221Z SIGABRT(6), PID: 1817873, Thread 1818313: 
2025-07-17T11:07:26.6862689Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6862892Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6863095Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6863292Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6863790Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6864064Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6864332Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6864336Z 
2025-07-17T11:07:26.6864481Z SIGABRT(6), PID: 1817867, Thread 1818121: 
2025-07-17T11:07:26.6864958Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6865226Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6865433Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6865626Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6866125Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6866333Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6866541Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6866546Z 
2025-07-17T11:07:26.6866633Z SIGABRT(6), PID: 1817872, Thread 1822168: 
2025-07-17T11:07:26.6867101Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6867308Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6867511Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6867716Z frame #3: <unknown function> + 0x9ca53 (0x7fdc95b96a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6867971Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5626a8d6dbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6868206Z frame #5: <unknown function> + 0x1d314f (0x5626a8e2c14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6868428Z frame #6: <unknown function> + 0x1d2f6b (0x5626a8e2bf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6868653Z frame #7: <unknown function> + 0x6df36 (0x5626a8cc6f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6868892Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6869121Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6869357Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6869584Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6869818Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6870047Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6870278Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6870504Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6870736Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6870959Z frame #17: <unknown function> + 0x147734 (0x5626a8da0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6871177Z frame #18: <unknown function> + 0x21ca82 (0x5626a8e75a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6871396Z frame #19: <unknown function> + 0x21c9e4 (0x5626a8e759e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6871607Z frame #20: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6871889Z frame #21: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6871957Z 
2025-07-17T11:07:26.6872057Z SIGABRT(6), PID: 1817869, Thread 1818008: 
2025-07-17T11:07:26.6872619Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6872843Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6873115Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6873323Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6873830Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6874041Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6874257Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6874350Z SIGABRT(6), PID: 1817871, Thread 1818112: 
2025-07-17T11:07:26.6874827Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6875032Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6875239Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6875438Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6875939Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6876148Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6876361Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6876365Z 
2025-07-17T11:07:26.6876370Z 
2025-07-17T11:07:26.6876458Z SIGABRT(6), PID: 1817873, Thread 1818316: 
2025-07-17T11:07:26.6876934Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6877138Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6877341Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6877536Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6878036Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6878240Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6878450Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6878454Z 
2025-07-17T11:07:26.6878543Z SIGABRT(6), PID: 1817866, Thread 1817878: 
2025-07-17T11:07:26.6879020Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6879228Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6879432Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6879627Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6880191Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6880519Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6880732Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6880736Z 
2025-07-17T11:07:26.6880885Z SIGABRT(6), PID: 1817867, Thread 1818125: 
2025-07-17T11:07:26.6881361Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6881565Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6881768Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6881963Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6882458Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6882669Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6882873Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6882879Z 
2025-07-17T11:07:26.6882968Z SIGABRT(6), PID: 1817873, Thread 1818318: 
2025-07-17T11:07:26.6883436Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6883640Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6883840Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6884041Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6884535Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6884741Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6884945Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6884950Z 
2025-07-17T11:07:26.6885041Z SIGABRT(6), PID: 1817869, Thread 1818011: 
2025-07-17T11:07:26.6885506Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6885713Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6885918Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6886115Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6886610Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6886817Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6887023Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6887027Z 
2025-07-17T11:07:26.6887117Z SIGABRT(6), PID: 1817872, Thread 1822181: 
2025-07-17T11:07:26.6887585Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6887862Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6888094Z frame #2: read + 0x4c (0x7fdc95c0e81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6888277Z frame #3: _Py_read + 0x4c (0x5626a8e33a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6888568Z frame #4: <unknown function> + 0x208c3b (0x5626a8e61c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6888795Z frame #5: <unknown function> + 0x14648d (0x5626a8d9f48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6889109Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5626a8dae215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6889332Z frame #7: <unknown function> + 0x208ae7 (0x5626a8e61ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6889545Z frame #8: <unknown function> + 0x208472 (0x5626a8e61472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6889765Z frame #9: <unknown function> + 0x1da6ee (0x5626a8e336ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6889994Z frame #10: <unknown function> + 0x1483bc (0x5626a8da13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6890241Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6890467Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6890705Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5626a8d853bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6890925Z frame #14: <unknown function> + 0x147734 (0x5626a8da0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6891174Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5626a8d87c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6891394Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6891631Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6891854Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5626a8d950ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6892092Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5626a8d857b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6892309Z frame #20: <unknown function> + 0x147734 (0x5626a8da0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6892534Z frame #21: <unknown function> + 0x21ca82 (0x5626a8e75a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6892753Z frame #22: <unknown function> + 0x21c9e4 (0x5626a8e759e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6892973Z frame #23: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6893187Z frame #24: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6893192Z 
2025-07-17T11:07:26.6893288Z SIGABRT(6), PID: 1817871, Thread 1818118: 
2025-07-17T11:07:26.6893766Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6893981Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6894188Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6894390Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6894895Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6895107Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6895312Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6895407Z SIGABRT(6), PID: 1817866, Thread 1817879: 
2025-07-17T11:07:26.6896036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6896335Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6896636Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6896854Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6897436Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6897649Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6897858Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6897863Z 
2025-07-17T11:07:26.6897871Z 
2025-07-17T11:07:26.6897963Z SIGABRT(6), PID: 1817867, Thread 1818130: 
2025-07-17T11:07:26.6898446Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6898651Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6898857Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6899054Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6899561Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6899765Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6899973Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6899979Z 
2025-07-17T11:07:26.6900067Z SIGABRT(6), PID: 1817870, Thread 1822246: 
2025-07-17T11:07:26.6900537Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6900738Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6900943Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6901140Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6901517Z frame #4: <unknown function> + 0x5a02023 (0x7f6af8693023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6902088Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6af86971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6902685Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6af868e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6903066Z frame #7: <unknown function> + 0x79d655 (0x7f6b09b7e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6903302Z frame #8: <unknown function> + 0xdc253 (0x7f6ac0bc9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6903506Z frame #9: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6903722Z frame #10: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6903811Z SIGABRT(6), PID: 1817873, Thread 1818319: 
2025-07-17T11:07:26.6904490Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6904778Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6905041Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6905242Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6905805Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6906008Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6906215Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6906219Z 
2025-07-17T11:07:26.6906223Z 
2025-07-17T11:07:26.6906310Z SIGABRT(6), PID: 1817869, Thread 1818014: 
2025-07-17T11:07:26.6906787Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6906994Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6907199Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6907396Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6907897Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6908105Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6908314Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6908319Z 
2025-07-17T11:07:26.6908407Z SIGABRT(6), PID: 1817866, Thread 1817880: 
2025-07-17T11:07:26.6908877Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6909082Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6909287Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6909484Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6909979Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6910189Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6910396Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6910402Z 
2025-07-17T11:07:26.6910492Z SIGABRT(6), PID: 1817867, Thread 1818134: 
2025-07-17T11:07:26.6910957Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6911163Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6911368Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6911566Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6912061Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6912344Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6912613Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6912617Z 
2025-07-17T11:07:26.6912710Z SIGABRT(6), PID: 1817871, Thread 1818122: 
2025-07-17T11:07:26.6913235Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6913504Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6913709Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6913911Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6914406Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6914615Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6914823Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6914827Z 
2025-07-17T11:07:26.6914920Z SIGABRT(6), PID: 1817873, Thread 1818321: 
2025-07-17T11:07:26.6915390Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6915597Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6915796Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6915990Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6916484Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6916691Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6916896Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6916901Z 
2025-07-17T11:07:26.6916991Z SIGABRT(6), PID: 1817869, Thread 1818017: 
2025-07-17T11:07:26.6917455Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6917662Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6917862Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6918059Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6918556Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6918766Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6918975Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6918982Z 
2025-07-17T11:07:26.6919067Z SIGABRT(6), PID: 1817868, Thread 1822232: 
2025-07-17T11:07:26.6919538Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6919739Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6919946Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6920145Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6920656Z frame #4: <unknown function> + 0x5a02023 (0x7f69d6aa8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6921279Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f69d6aac1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6921878Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f69d6aa31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6922321Z frame #7: <unknown function> + 0x79d655 (0x7f69e7f8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6922558Z frame #8: <unknown function> + 0xdc253 (0x7f699efde253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6922764Z frame #9: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6922978Z frame #10: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6922982Z 
2025-07-17T11:07:26.6923070Z SIGABRT(6), PID: 1817867, Thread 1818137: 
2025-07-17T11:07:26.6923547Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6923753Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6923963Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6924160Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6924657Z frame #4: <unknown function> + 0x3475ab (0x7f13e26995ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6924865Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6925076Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6925080Z 
2025-07-17T11:07:26.6925169Z SIGABRT(6), PID: 1817871, Thread 1818126: 
2025-07-17T11:07:26.6925645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6925849Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6926057Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6926255Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6926759Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6926965Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6927177Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6927181Z 
2025-07-17T11:07:26.6927268Z SIGABRT(6), PID: 1817873, Thread 1818324: 
2025-07-17T11:07:26.6927736Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6927940Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6928139Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6928334Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6928900Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6929167Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6929435Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6929530Z SIGABRT(6), PID: 1817866, Thread 1817881: 
2025-07-17T11:07:26.6930064Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6930273Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6930476Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6930677Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6931171Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6931379Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6931583Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6931589Z 
2025-07-17T11:07:26.6931592Z 
2025-07-17T11:07:26.6931682Z SIGABRT(6), PID: 1817869, Thread 1818019: 
2025-07-17T11:07:26.6932146Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6932353Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6932552Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6932751Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6933249Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6933455Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6933659Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6933667Z 
2025-07-17T11:07:26.6933755Z SIGABRT(6), PID: 1817867, Thread 1818394: 
2025-07-17T11:07:26.6934219Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6934676Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f12f3ef808a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6934885Z frame #2: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6935077Z frame #3: pthread_kill + 0x12c (0x7f13e58039fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6935243Z frame #4: raise + 0x16 (0x7f13e57af476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6935402Z frame #5: abort + 0xd3 (0x7f13e57957f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6935638Z frame #6: <unknown function> + 0x972cc (0x7f1261a812cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6935872Z frame #7: <unknown function> + 0x95f47 (0x7f1261a7ff47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6936180Z frame #8: <unknown function> + 0x34b71 (0x7f1261a1eb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.6936389Z frame #9: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6936713Z frame #10: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6936787Z 
2025-07-17T11:07:26.6936884Z SIGABRT(6), PID: 1817873, Thread 1818349: 
2025-07-17T11:07:26.6937427Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6937643Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6937917Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6938118Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6938617Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6938823Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6939029Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6939121Z SIGABRT(6), PID: 1817871, Thread 1818129: 
2025-07-17T11:07:26.6939591Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6939799Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6940002Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6940200Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6940693Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6940906Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6941111Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6941115Z 
2025-07-17T11:07:26.6941121Z 
2025-07-17T11:07:26.6941210Z SIGABRT(6), PID: 1817869, Thread 1818022: 
2025-07-17T11:07:26.6941682Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6941889Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6942090Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6942284Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6942783Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6942990Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6943198Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6943204Z 
2025-07-17T11:07:26.6943290Z SIGABRT(6), PID: 1817866, Thread 1817882: 
2025-07-17T11:07:26.6943760Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6943966Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6944169Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6944362Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6944926Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6945195Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6945466Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6945471Z 
2025-07-17T11:07:26.6945563Z SIGABRT(6), PID: 1817867, Thread 1820911: 
2025-07-17T11:07:26.6946098Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6946302Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6946468Z frame #2: __poll + 0x4f (0x7f13e5885bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6946875Z frame #3: <unknown function> + 0x489f (0x7f13e539189f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.6947104Z frame #4: <unknown function> + 0x1483bc (0x56497497f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6947345Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6947573Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6947805Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6948031Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6948273Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x564974967a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6948502Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6948747Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x564974965c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6948974Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6949208Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6949430Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6949668Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6949891Z frame #16: <unknown function> + 0x147734 (0x56497497e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6950108Z frame #17: <unknown function> + 0x21ca82 (0x564974a53a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6950327Z frame #18: <unknown function> + 0x21c9e4 (0x564974a539e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.6950534Z frame #19: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6950745Z frame #20: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6950751Z 
2025-07-17T11:07:26.6950840Z SIGABRT(6), PID: 1817873, Thread 1818350: 
2025-07-17T11:07:26.6951315Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6951525Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6951734Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6951930Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6952432Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6952707Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6952978Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6952982Z 
2025-07-17T11:07:26.6953128Z SIGABRT(6), PID: 1817871, Thread 1818133: 
2025-07-17T11:07:26.6953606Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6953872Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6959424Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6959670Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6960217Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6960444Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6960664Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6960672Z 
2025-07-17T11:07:26.6960772Z SIGABRT(6), PID: 1817869, Thread 1818025: 
2025-07-17T11:07:26.6961271Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6961490Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6961700Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6961902Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6962420Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6962633Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6962847Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6962852Z 
2025-07-17T11:07:26.6962949Z SIGABRT(6), PID: 1817866, Thread 1817883: 
2025-07-17T11:07:26.6963430Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6963638Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6963842Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6964042Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6964538Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6964751Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6964959Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6964963Z 
2025-07-17T11:07:26.6965057Z SIGABRT(6), PID: 1817871, Thread 1818136: 
2025-07-17T11:07:26.6965525Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6965734Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6965938Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6966250Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6966818Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6967090Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6967308Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6967395Z 
2025-07-17T11:07:26.6967491Z SIGABRT(6), PID: 1817869, Thread 1818027: 
2025-07-17T11:07:26.6967959Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6968169Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6968376Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6968582Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6969076Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6969289Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6969498Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6969590Z SIGABRT(6), PID: 1817873, Thread 1818351: 
2025-07-17T11:07:26.6970058Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6970266Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6970468Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6970671Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6971167Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6971374Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6971581Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6971586Z 
2025-07-17T11:07:26.6971590Z 
2025-07-17T11:07:26.6971680Z SIGABRT(6), PID: 1817866, Thread 1817884: 
2025-07-17T11:07:26.6972147Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6972357Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6972563Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6972764Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6973266Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6973482Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6973694Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6973700Z 
2025-07-17T11:07:26.6973789Z SIGABRT(6), PID: 1817869, Thread 1818030: 
2025-07-17T11:07:26.6974253Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6974530Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6974799Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6975055Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6975553Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6975823Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6976112Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6976204Z SIGABRT(6), PID: 1817871, Thread 1818140: 
2025-07-17T11:07:26.6976672Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6976880Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6977086Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6977282Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6977777Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6977985Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6978197Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6978202Z 
2025-07-17T11:07:26.6978206Z 
2025-07-17T11:07:26.6978292Z SIGABRT(6), PID: 1817870, Thread 1822248: 
2025-07-17T11:07:26.6978764Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6ac295cb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6978970Z frame #1: <unknown function> + 0x42520 (0x7f6bb4215520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6979177Z frame #2: <unknown function> + 0x91117 (0x7f6bb4264117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6979372Z frame #3: pthread_cond_wait + 0x211 (0x7f6bb4266a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6979756Z frame #4: <unknown function> + 0x5a02023 (0x7f6af8693023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6980324Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6af86971ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6980924Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6af868e1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6981309Z frame #7: <unknown function> + 0x79d655 (0x7f6b09b7e655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6981545Z frame #8: <unknown function> + 0xdc253 (0x7f6ac0bc9253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6981754Z frame #9: <unknown function> + 0x94ac3 (0x7f6bb4267ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6981973Z frame #10: <unknown function> + 0x126850 (0x7f6bb42f9850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6981977Z 
2025-07-17T11:07:26.6982068Z SIGABRT(6), PID: 1817872, Thread 1822220: 
2025-07-17T11:07:26.6982540Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6982839Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6983143Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6983417Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6983794Z frame #4: <unknown function> + 0x5a02023 (0x7fdbd9fbb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6984434Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fdbd9fbf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6985030Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fdbd9fb61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.6985416Z frame #7: <unknown function> + 0x79d655 (0x7fdbeb4a2655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.6985650Z frame #8: <unknown function> + 0xdc253 (0x7fdba24f1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6985861Z frame #9: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6986077Z frame #10: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6986083Z 
2025-07-17T11:07:26.6986174Z SIGABRT(6), PID: 1817873, Thread 1818352: 
2025-07-17T11:07:26.6986641Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6986848Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6987048Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6987249Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6987754Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6987962Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6988170Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6988176Z 
2025-07-17T11:07:26.6988267Z SIGABRT(6), PID: 1817866, Thread 1817885: 
2025-07-17T11:07:26.6988732Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6988940Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6989146Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6989346Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6989843Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6990052Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6990260Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6990265Z 
2025-07-17T11:07:26.6990355Z SIGABRT(6), PID: 1817867, Thread 1822119: 
2025-07-17T11:07:26.6990819Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6991027Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6991367Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6991595Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f13e58012dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6992098Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f131fc40048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6992584Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f131fc42bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.6992816Z frame #6: <unknown function> + 0xdc253 (0x7f12f2164253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.6993024Z frame #7: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6993233Z frame #8: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6993240Z 
2025-07-17T11:07:26.6993334Z SIGABRT(6), PID: 1817869, Thread 1818033: 
2025-07-17T11:07:26.6993803Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6994010Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6994216Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6994414Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6994915Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6995121Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6995331Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6995337Z 
2025-07-17T11:07:26.6995425Z SIGABRT(6), PID: 1817871, Thread 1818142: 
2025-07-17T11:07:26.6995895Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6996099Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6996307Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6996505Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6997003Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6997210Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6997426Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6997430Z 
2025-07-17T11:07:26.6997517Z SIGABRT(6), PID: 1817873, Thread 1818353: 
2025-07-17T11:07:26.6997982Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.6998186Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6998391Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6998585Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6999079Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.6999358Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6999631Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.6999636Z 
2025-07-17T11:07:26.6999815Z SIGABRT(6), PID: 1817868, Thread 1822233: 
2025-07-17T11:07:26.7000286Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7000553Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7000760Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7000957Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7001332Z frame #4: <unknown function> + 0x5a02023 (0x7f69d6aa8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7001900Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f69d6aac1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7002503Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f69d6aa31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7002883Z frame #7: <unknown function> + 0x79d655 (0x7f69e7f8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7003114Z frame #8: <unknown function> + 0xdc253 (0x7f699efde253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7003321Z frame #9: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7003536Z frame #10: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7003542Z 
2025-07-17T11:07:26.7003635Z SIGABRT(6), PID: 1817866, Thread 1817886: 
2025-07-17T11:07:26.7004105Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7004312Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7004514Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7004716Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7005209Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7005421Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7005631Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7005637Z 
2025-07-17T11:07:26.7005727Z SIGABRT(6), PID: 1817869, Thread 1818036: 
2025-07-17T11:07:26.7006193Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7006399Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7006603Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7006804Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7007301Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7007582Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7007854Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7007858Z 
2025-07-17T11:07:26.7007949Z SIGABRT(6), PID: 1817871, Thread 1818144: 
2025-07-17T11:07:26.7008474Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7008744Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7008948Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7009149Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7009644Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7009855Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7010065Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7010069Z 
2025-07-17T11:07:26.7010163Z SIGABRT(6), PID: 1817867, Thread 1822120: 
2025-07-17T11:07:26.7010623Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7010831Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7011034Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7011255Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f13e58012dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7011728Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f131fc3a741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7011959Z frame #5: <unknown function> + 0xdc253 (0x7f12f2164253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7012161Z frame #6: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7012372Z frame #7: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7012376Z 
2025-07-17T11:07:26.7012466Z SIGABRT(6), PID: 1817873, Thread 1818354: 
2025-07-17T11:07:26.7012938Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7013146Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7013350Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7013550Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7014052Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7014259Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7014465Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7014471Z 
2025-07-17T11:07:26.7014563Z SIGABRT(6), PID: 1817866, Thread 1817887: 
2025-07-17T11:07:26.7015028Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7015238Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7015514Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7015774Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7016389Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7016605Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7016893Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7016984Z SIGABRT(6), PID: 1817869, Thread 1818039: 
2025-07-17T11:07:26.7017445Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7017651Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7017855Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7018055Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7018550Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7018758Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7018967Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7018972Z 
2025-07-17T11:07:26.7018975Z 
2025-07-17T11:07:26.7019063Z SIGABRT(6), PID: 1817867, Thread 1822121: 
2025-07-17T11:07:26.7019520Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7019738Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7019905Z frame #2: ioctl + 0x3f (0x7f13e588794f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7020144Z frame #3: <unknown function> + 0x13eef0 (0x7f1261b28ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7020374Z frame #4: <unknown function> + 0x1378c4 (0x7f1261b218c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7020609Z frame #5: <unknown function> + 0x95b2b (0x7f1261a7fb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7020834Z frame #6: <unknown function> + 0x34b71 (0x7f1261a1eb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7021043Z frame #7: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7021249Z frame #8: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7021253Z 
2025-07-17T11:07:26.7021345Z SIGABRT(6), PID: 1817871, Thread 1818146: 
2025-07-17T11:07:26.7021808Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7022023Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7022229Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7022430Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7022941Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7023156Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7023362Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7023366Z 
2025-07-17T11:07:26.7023545Z SIGABRT(6), PID: 1817873, Thread 1818355: 
2025-07-17T11:07:26.7024079Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7024347Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7024552Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7024812Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7025303Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7025508Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7025717Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7025725Z 
2025-07-17T11:07:26.7025813Z SIGABRT(6), PID: 1817866, Thread 1817888: 
2025-07-17T11:07:26.7026282Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7026487Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7026694Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7026890Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7027380Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7027586Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7027797Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7027889Z SIGABRT(6), PID: 1817869, Thread 1818042: 
2025-07-17T11:07:26.7028354Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7028557Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7028764Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7028960Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7029453Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7029659Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7029870Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7029874Z 
2025-07-17T11:07:26.7029878Z 
2025-07-17T11:07:26.7029964Z SIGABRT(6), PID: 1817867, Thread 1822147: 
2025-07-17T11:07:26.7030427Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7030632Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7030800Z frame #2: __poll + 0x4f (0x7f13e5885bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7031011Z frame #3: <unknown function> + 0xd4540 (0x7f12a9c98540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7031217Z frame #4: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7031497Z frame #5: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7031580Z 
2025-07-17T11:07:26.7031674Z SIGABRT(6), PID: 1817871, Thread 1818148: 
2025-07-17T11:07:26.7032196Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7032408Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7032781Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7032983Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7033474Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7033683Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7033901Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7033907Z 
2025-07-17T11:07:26.7034009Z SIGABRT(6), PID: 1817873, Thread 1818356: 
2025-07-17T11:07:26.7034474Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7034680Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7034883Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7035079Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7035569Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7035780Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7035996Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7036003Z 
2025-07-17T11:07:26.7036092Z SIGABRT(6), PID: 1817867, Thread 1822155: 
2025-07-17T11:07:26.7036558Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7036771Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7036935Z frame #2: __poll + 0x4f (0x7f13e5885bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7037146Z frame #3: <unknown function> + 0xd65a2 (0x7f12a9c9a5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7037354Z frame #4: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7037561Z frame #5: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7037653Z SIGABRT(6), PID: 1817866, Thread 1817889: 
2025-07-17T11:07:26.7038117Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7038324Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7038527Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7038724Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7039212Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7039419Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7039703Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7039765Z 
2025-07-17T11:07:26.7039768Z 
2025-07-17T11:07:26.7039863Z SIGABRT(6), PID: 1817869, Thread 1818045: 
2025-07-17T11:07:26.7040394Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7040606Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7040872Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7041074Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7041565Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7041775Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7041982Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7041986Z 
2025-07-17T11:07:26.7042077Z SIGABRT(6), PID: 1817867, Thread 1822161: 
2025-07-17T11:07:26.7042538Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7042746Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7042947Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7043142Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7043352Z frame #4: <unknown function> + 0xd1e7f (0x7f12a9c95e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7043558Z frame #5: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7043763Z frame #6: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7043853Z SIGABRT(6), PID: 1817871, Thread 1818150: 
2025-07-17T11:07:26.7044317Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7044524Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7044726Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7044926Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7045417Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7045629Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7045835Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7045839Z 
2025-07-17T11:07:26.7045844Z 
2025-07-17T11:07:26.7045935Z SIGABRT(6), PID: 1817873, Thread 1818357: 
2025-07-17T11:07:26.7046397Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7046606Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7046806Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7047005Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7047572Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7047842Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7048106Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7048114Z 
2025-07-17T11:07:26.7048204Z SIGABRT(6), PID: 1817872, Thread 1822221: 
2025-07-17T11:07:26.7048732Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7048938Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7049146Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7049343Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7049723Z frame #4: <unknown function> + 0x5a02023 (0x7fdbd9fbb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7050293Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fdbd9fbf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7050890Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fdbd9fb61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7051274Z frame #7: <unknown function> + 0x79d655 (0x7fdbeb4a2655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7051505Z frame #8: <unknown function> + 0xdc253 (0x7fdba24f1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7051712Z frame #9: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7051938Z frame #10: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7051942Z 
2025-07-17T11:07:26.7052029Z SIGABRT(6), PID: 1817866, Thread 1817890: 
2025-07-17T11:07:26.7052503Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7052712Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7052921Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7053119Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7053619Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7053827Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7054038Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7054128Z SIGABRT(6), PID: 1817869, Thread 1818048: 
2025-07-17T11:07:26.7054594Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7054802Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7055006Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7055204Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7055773Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7056125Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7056340Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7056421Z 
2025-07-17T11:07:26.7056426Z 
2025-07-17T11:07:26.7056518Z SIGABRT(6), PID: 1817871, Thread 1818152: 
2025-07-17T11:07:26.7056984Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7057267Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7057474Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7057673Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7058171Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7058385Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7058594Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7058598Z 
2025-07-17T11:07:26.7058689Z SIGABRT(6), PID: 1817867, Thread 1822167: 
2025-07-17T11:07:26.7059155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7059361Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7059562Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7059774Z frame #3: <unknown function> + 0x9ca53 (0x7f13e5809a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7060031Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x56497494bbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7060261Z frame #5: <unknown function> + 0x1d314f (0x564974a0a14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7060485Z frame #6: <unknown function> + 0x1d2f6b (0x564974a09f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7060705Z frame #7: <unknown function> + 0x6df36 (0x5649748a4f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7060973Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7061199Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7061435Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7061662Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7061900Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7062129Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7062363Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7062586Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7062816Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7063036Z frame #17: <unknown function> + 0x147734 (0x56497497e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7063262Z frame #18: <unknown function> + 0x21ca82 (0x564974a53a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7063482Z frame #19: <unknown function> + 0x21c9e4 (0x564974a539e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7063818Z frame #20: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7064114Z frame #21: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7064267Z SIGABRT(6), PID: 1817873, Thread 1818358: 
2025-07-17T11:07:26.7064745Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7065020Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7065225Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7065424Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7065935Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7066145Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7066353Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7066360Z 
2025-07-17T11:07:26.7066364Z 
2025-07-17T11:07:26.7066452Z SIGABRT(6), PID: 1817866, Thread 1817891: 
2025-07-17T11:07:26.7066931Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7067143Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7067352Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7067554Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7068061Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7068270Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7068481Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7068485Z 
2025-07-17T11:07:26.7068574Z SIGABRT(6), PID: 1817869, Thread 1818050: 
2025-07-17T11:07:26.7069047Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7069253Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7069461Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7069659Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7070166Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7070374Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7070587Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7070593Z 
2025-07-17T11:07:26.7070680Z SIGABRT(6), PID: 1817871, Thread 1818154: 
2025-07-17T11:07:26.7071150Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7071353Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7071559Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7071827Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7072441Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7072657Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7072873Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7072935Z 
2025-07-17T11:07:26.7073027Z SIGABRT(6), PID: 1817873, Thread 1818359: 
2025-07-17T11:07:26.7073496Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7073702Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7073905Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7074105Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7074599Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7074805Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7075012Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7075017Z 
2025-07-17T11:07:26.7075108Z SIGABRT(6), PID: 1817868, Thread 1822234: 
2025-07-17T11:07:26.7075577Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7075785Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7075990Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7076191Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7076567Z frame #4: <unknown function> + 0x5a02023 (0x7f69d6aa8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7077133Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f69d6aac1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7077731Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f69d6aa31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7078115Z frame #7: <unknown function> + 0x79d655 (0x7f69e7f8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7078347Z frame #8: <unknown function> + 0xdc253 (0x7f699efde253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7078559Z frame #9: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7078771Z frame #10: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7078776Z 
2025-07-17T11:07:26.7078866Z SIGABRT(6), PID: 1817866, Thread 1817892: 
2025-07-17T11:07:26.7079335Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7079546Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7079751Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7080022Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7080578Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7080854Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7081065Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7081218Z SIGABRT(6), PID: 1817867, Thread 1822179: 
2025-07-17T11:07:26.7081685Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7081893Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7082057Z frame #2: read + 0x4c (0x7f13e588181c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7082245Z frame #3: _Py_read + 0x4c (0x564974a11a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7082475Z frame #4: <unknown function> + 0x208c3b (0x564974a3fc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7082698Z frame #5: <unknown function> + 0x14648d (0x56497497d48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7082942Z frame #6: PyObject_VectorcallMethod + 0x85 (0x56497498c215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7083166Z frame #7: <unknown function> + 0x208ae7 (0x564974a3fae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7083381Z frame #8: <unknown function> + 0x208472 (0x564974a3f472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7083601Z frame #9: <unknown function> + 0x1da6ee (0x564974a116ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7083824Z frame #10: <unknown function> + 0x1483bc (0x56497497f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7084069Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7084294Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7084530Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5649749633bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7084746Z frame #14: <unknown function> + 0x147734 (0x56497497e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7084994Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x564974965c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7085215Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7085447Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7085668Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5649749730ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7085901Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5649749637b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7086117Z frame #20: <unknown function> + 0x147734 (0x56497497e734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7086337Z frame #21: <unknown function> + 0x21ca82 (0x564974a53a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7086558Z frame #22: <unknown function> + 0x21c9e4 (0x564974a539e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7086772Z frame #23: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7086986Z frame #24: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7087081Z SIGABRT(6), PID: 1817869, Thread 1818053: 
2025-07-17T11:07:26.7087549Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7087763Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7088041Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7088304Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7088865Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7089163Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7089375Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7089379Z 
2025-07-17T11:07:26.7089388Z 
2025-07-17T11:07:26.7089392Z 
2025-07-17T11:07:26.7089483Z SIGABRT(6), PID: 1817871, Thread 1818156: 
2025-07-17T11:07:26.7089952Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7090160Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7090367Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7090566Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7091063Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7091271Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7091482Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7091486Z 
2025-07-17T11:07:26.7091575Z SIGABRT(6), PID: 1817873, Thread 1818360: 
2025-07-17T11:07:26.7092047Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7092252Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7092457Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7092652Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7093147Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7093350Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7093557Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7093562Z 
2025-07-17T11:07:26.7093649Z SIGABRT(6), PID: 1817866, Thread 1817893: 
2025-07-17T11:07:26.7094120Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7094329Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7094536Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7094730Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7095229Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7095433Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7095640Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7095644Z 
2025-07-17T11:07:26.7095808Z SIGABRT(6), PID: 1817869, Thread 1818056: 
2025-07-17T11:07:26.7096418Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7096706Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7096919Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7097190Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7097688Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7097895Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7098107Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7098113Z 
2025-07-17T11:07:26.7098201Z SIGABRT(6), PID: 1817873, Thread 1818361: 
2025-07-17T11:07:26.7098670Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7098873Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7099076Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7099269Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7099759Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7099965Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7100170Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7100176Z 
2025-07-17T11:07:26.7100267Z SIGABRT(6), PID: 1817871, Thread 1818158: 
2025-07-17T11:07:26.7100731Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7100937Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7101141Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7101340Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7101830Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7102040Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7102246Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7102250Z 
2025-07-17T11:07:26.7102340Z SIGABRT(6), PID: 1817869, Thread 1818059: 
2025-07-17T11:07:26.7102804Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7103010Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7103212Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7103407Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7103982Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7104263Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7104471Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7104532Z 
2025-07-17T11:07:26.7104625Z SIGABRT(6), PID: 1817866, Thread 1817894: 
2025-07-17T11:07:26.7105091Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7105360Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7105566Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7105765Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7106261Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7106470Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7106676Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7106680Z 
2025-07-17T11:07:26.7106771Z SIGABRT(6), PID: 1817873, Thread 1818362: 
2025-07-17T11:07:26.7107237Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7107444Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7107642Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7107839Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7108328Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7108537Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7108745Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7108750Z 
2025-07-17T11:07:26.7108841Z SIGABRT(6), PID: 1817871, Thread 1818160: 
2025-07-17T11:07:26.7109300Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7109507Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7109710Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7109912Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7110409Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7110617Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7110829Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7110835Z 
2025-07-17T11:07:26.7110923Z SIGABRT(6), PID: 1817869, Thread 1818062: 
2025-07-17T11:07:26.7111389Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7111593Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7111797Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7112123Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7112676Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7112886Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7113155Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7113158Z 
2025-07-17T11:07:26.7113247Z SIGABRT(6), PID: 1817873, Thread 1818363: 
2025-07-17T11:07:26.7113715Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7113919Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7114128Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7114325Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7114822Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7115026Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7115242Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7115331Z SIGABRT(6), PID: 1817866, Thread 1817896: 
2025-07-17T11:07:26.7115798Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7116006Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7116215Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7116413Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7116909Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7117117Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7117325Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7117329Z 
2025-07-17T11:07:26.7117333Z 
2025-07-17T11:07:26.7117420Z SIGABRT(6), PID: 1817872, Thread 1822222: 
2025-07-17T11:07:26.7117887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7118096Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7118303Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7118502Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7118881Z frame #4: <unknown function> + 0x5a02023 (0x7fdbd9fbb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7119449Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fdbd9fbf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7120044Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fdbd9fb61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7120499Z frame #7: <unknown function> + 0x79d655 (0x7fdbeb4a2655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7120796Z frame #8: <unknown function> + 0xdc253 (0x7fdba24f1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7121090Z frame #9: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7121308Z frame #10: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7121374Z 
2025-07-17T11:07:26.7121467Z SIGABRT(6), PID: 1817871, Thread 1818162: 
2025-07-17T11:07:26.7121933Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7122144Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7122349Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7122551Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7123050Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7123259Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7123468Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7123472Z 
2025-07-17T11:07:26.7123562Z SIGABRT(6), PID: 1817869, Thread 1818065: 
2025-07-17T11:07:26.7124023Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7124231Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7124435Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7124636Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7125135Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7125343Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7125554Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7125559Z 
2025-07-17T11:07:26.7125649Z SIGABRT(6), PID: 1817873, Thread 1818364: 
2025-07-17T11:07:26.7126111Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7126318Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7126519Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7126715Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7127204Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7127416Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7127620Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7127623Z 
2025-07-17T11:07:26.7127716Z SIGABRT(6), PID: 1817866, Thread 1817897: 
2025-07-17T11:07:26.7128248Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7128514Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7128720Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7128985Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7129480Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7129751Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7129956Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7129963Z 
2025-07-17T11:07:26.7130053Z SIGABRT(6), PID: 1817871, Thread 1818164: 
2025-07-17T11:07:26.7130520Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7130725Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7130932Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7131129Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7131623Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7131830Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7132041Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7132130Z SIGABRT(6), PID: 1817869, Thread 1818068: 
2025-07-17T11:07:26.7132600Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7132806Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7133012Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7133209Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7133700Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7133907Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7134120Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7134124Z 
2025-07-17T11:07:26.7134128Z 
2025-07-17T11:07:26.7134216Z SIGABRT(6), PID: 1817873, Thread 1818365: 
2025-07-17T11:07:26.7134683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7134888Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7135092Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7135288Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7135778Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7136043Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7136338Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7136412Z 
2025-07-17T11:07:26.7136505Z SIGABRT(6), PID: 1817866, Thread 1817899: 
2025-07-17T11:07:26.7137049Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7137258Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7137556Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7137752Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7138252Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7138457Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7138669Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7138674Z 
2025-07-17T11:07:26.7138763Z SIGABRT(6), PID: 1817868, Thread 1822235: 
2025-07-17T11:07:26.7139241Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f69a0d71b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7139447Z frame #1: <unknown function> + 0x42520 (0x7f6a92629520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7139650Z frame #2: <unknown function> + 0x91117 (0x7f6a92678117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7139845Z frame #3: pthread_cond_wait + 0x211 (0x7f6a9267aa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7140220Z frame #4: <unknown function> + 0x5a02023 (0x7f69d6aa8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7140786Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f69d6aac1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7141383Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f69d6aa31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7141765Z frame #7: <unknown function> + 0x79d655 (0x7f69e7f8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7141999Z frame #8: <unknown function> + 0xdc253 (0x7f699efde253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7142209Z frame #9: <unknown function> + 0x94ac3 (0x7f6a9267bac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7142423Z frame #10: <unknown function> + 0x126850 (0x7f6a9270d850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7142515Z SIGABRT(6), PID: 1817869, Thread 1818071: 
2025-07-17T11:07:26.7142984Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7143196Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7143401Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7143601Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7144099Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7144310Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7144521Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7144525Z 
2025-07-17T11:07:26.7144529Z 
2025-07-17T11:07:26.7144757Z SIGABRT(6), PID: 1817871, Thread 1818167: 
2025-07-17T11:07:26.7145229Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7145496Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7145705Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7145965Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7146463Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7146672Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7146885Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7146891Z 
2025-07-17T11:07:26.7146983Z SIGABRT(6), PID: 1817873, Thread 1818366: 
2025-07-17T11:07:26.7147449Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7147654Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7147855Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7148049Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7148540Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7148746Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7148955Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7148959Z 
2025-07-17T11:07:26.7149049Z SIGABRT(6), PID: 1817866, Thread 1817900: 
2025-07-17T11:07:26.7149516Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7149723Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7149925Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7150122Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7150614Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7150827Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7151034Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7151041Z 
2025-07-17T11:07:26.7151130Z SIGABRT(6), PID: 1817867, Thread 1822212: 
2025-07-17T11:07:26.7151597Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7151802Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7152004Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7152200Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7152571Z frame #4: <unknown function> + 0x5a02023 (0x7f1329c2e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7153283Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1329c321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7153940Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1329c291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7154378Z frame #7: <unknown function> + 0x79d655 (0x7f133b115655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7154609Z frame #8: <unknown function> + 0xdc253 (0x7f12f2164253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7154817Z frame #9: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7155034Z frame #10: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7155043Z 
2025-07-17T11:07:26.7155132Z SIGABRT(6), PID: 1817869, Thread 1818074: 
2025-07-17T11:07:26.7155603Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7155808Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7156011Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7156209Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7156713Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7156918Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7157127Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7157132Z 
2025-07-17T11:07:26.7157219Z SIGABRT(6), PID: 1817871, Thread 1818170: 
2025-07-17T11:07:26.7157690Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7157895Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7158105Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7158303Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7158796Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7159005Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7159218Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7159222Z 
2025-07-17T11:07:26.7159311Z SIGABRT(6), PID: 1817873, Thread 1818367: 
2025-07-17T11:07:26.7159777Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7159981Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7160184Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7160380Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7160871Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7161146Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7161415Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7161674Z SIGABRT(6), PID: 1817866, Thread 1817902: 
2025-07-17T11:07:26.7162148Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7162419Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7162623Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7162825Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7163320Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7163531Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7163743Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7163747Z 
2025-07-17T11:07:26.7163750Z 
2025-07-17T11:07:26.7163841Z SIGABRT(6), PID: 1817869, Thread 1818077: 
2025-07-17T11:07:26.7164302Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7164516Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7164718Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7164916Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7165412Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7165621Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7165831Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7165834Z 
2025-07-17T11:07:26.7165928Z SIGABRT(6), PID: 1817871, Thread 1818173: 
2025-07-17T11:07:26.7166392Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7166598Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7166802Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7167014Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7167513Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7167729Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7167937Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7167943Z 
2025-07-17T11:07:26.7168034Z SIGABRT(6), PID: 1817866, Thread 1817904: 
2025-07-17T11:07:26.7168503Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7168715Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7168920Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7169192Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7169803Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7170015Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7170222Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7170289Z 
2025-07-17T11:07:26.7170385Z SIGABRT(6), PID: 1817873, Thread 1818368: 
2025-07-17T11:07:26.7170849Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7171059Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7171260Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7171464Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7171960Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7172167Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7172377Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7172382Z 
2025-07-17T11:07:26.7172471Z SIGABRT(6), PID: 1817869, Thread 1818079: 
2025-07-17T11:07:26.7172940Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7173144Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7173355Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7173552Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7174050Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7174258Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7174469Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7174473Z 
2025-07-17T11:07:26.7174561Z SIGABRT(6), PID: 1817871, Thread 1818177: 
2025-07-17T11:07:26.7175025Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7175231Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7175438Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7175634Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7176199Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7176409Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7176615Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7176619Z 
2025-07-17T11:07:26.7176706Z SIGABRT(6), PID: 1817866, Thread 1817905: 
2025-07-17T11:07:26.7177263Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7177540Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7177816Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7178016Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7178510Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7178795Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7179006Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7179010Z 
2025-07-17T11:07:26.7179098Z SIGABRT(6), PID: 1817872, Thread 1822223: 
2025-07-17T11:07:26.7179568Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7179778Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7179986Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7180181Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7180562Z frame #4: <unknown function> + 0x5a02023 (0x7fdbd9fbb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7181129Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fdbd9fbf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7181727Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fdbd9fb61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7182112Z frame #7: <unknown function> + 0x79d655 (0x7fdbeb4a2655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7182349Z frame #8: <unknown function> + 0xdc253 (0x7fdba24f1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7182560Z frame #9: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7182777Z frame #10: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7182782Z 
2025-07-17T11:07:26.7182872Z SIGABRT(6), PID: 1817873, Thread 1818369: 
2025-07-17T11:07:26.7183340Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7183547Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7183750Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7183951Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7184447Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7184658Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7184862Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7184866Z 
2025-07-17T11:07:26.7184957Z SIGABRT(6), PID: 1817871, Thread 1818180: 
2025-07-17T11:07:26.7185424Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7185721Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7185985Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7186250Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7186747Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7187020Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7187229Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7187323Z SIGABRT(6), PID: 1817869, Thread 1818082: 
2025-07-17T11:07:26.7187786Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7187999Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7188201Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7188400Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7188895Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7189104Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7189312Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7189319Z 
2025-07-17T11:07:26.7189322Z 
2025-07-17T11:07:26.7189414Z SIGABRT(6), PID: 1817866, Thread 1817907: 
2025-07-17T11:07:26.7189884Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7190092Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7190296Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7190497Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7190993Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7191202Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7191409Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7191416Z 
2025-07-17T11:07:26.7191505Z SIGABRT(6), PID: 1817873, Thread 1818370: 
2025-07-17T11:07:26.7191972Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7192178Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7192383Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7192578Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7193071Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7193275Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7193483Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7193614Z 
2025-07-17T11:07:26.7193708Z SIGABRT(6), PID: 1817871, Thread 1818184: 
2025-07-17T11:07:26.7194240Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7194449Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7194654Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7194910Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7195406Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7195616Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7195828Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7195833Z 
2025-07-17T11:07:26.7195920Z SIGABRT(6), PID: 1817869, Thread 1818085: 
2025-07-17T11:07:26.7196391Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7196597Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7196803Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7196999Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7197498Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7197707Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7197918Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7197922Z 
2025-07-17T11:07:26.7198008Z SIGABRT(6), PID: 1817866, Thread 1817909: 
2025-07-17T11:07:26.7198477Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7198683Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7198888Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7199083Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7199579Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7199786Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7199998Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7200002Z 
2025-07-17T11:07:26.7200092Z SIGABRT(6), PID: 1817873, Thread 1818371: 
2025-07-17T11:07:26.7200559Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7200765Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7200967Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7201160Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7201726Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7201990Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7202263Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7202267Z 
2025-07-17T11:07:26.7202359Z SIGABRT(6), PID: 1817869, Thread 1818088: 
2025-07-17T11:07:26.7202825Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7203096Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7203298Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7203498Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7203991Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7204199Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7204407Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7204501Z SIGABRT(6), PID: 1817871, Thread 1818187: 
2025-07-17T11:07:26.7204969Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7205176Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7205377Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7205577Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7206068Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7206279Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7206486Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7206492Z 
2025-07-17T11:07:26.7206496Z 
2025-07-17T11:07:26.7206589Z SIGABRT(6), PID: 1817866, Thread 1817912: 
2025-07-17T11:07:26.7207055Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7207263Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7207464Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7207669Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7208161Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7208371Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7208581Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7208586Z 
2025-07-17T11:07:26.7208680Z SIGABRT(6), PID: 1817873, Thread 1818372: 
2025-07-17T11:07:26.7209143Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7209348Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7209615Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7209886Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7210440Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7210649Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7210937Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7210942Z 
2025-07-17T11:07:26.7211036Z SIGABRT(6), PID: 1817869, Thread 1818091: 
2025-07-17T11:07:26.7211506Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7211714Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7211918Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7212116Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7212616Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7212828Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7213034Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7213040Z 
2025-07-17T11:07:26.7213132Z SIGABRT(6), PID: 1817871, Thread 1818191: 
2025-07-17T11:07:26.7213601Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7213807Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7214016Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7214215Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7214709Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7214918Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7215135Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7215139Z 
2025-07-17T11:07:26.7215228Z SIGABRT(6), PID: 1817867, Thread 1822213: 
2025-07-17T11:07:26.7215700Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7215986Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7216200Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7216399Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7216774Z frame #4: <unknown function> + 0x5a02023 (0x7f1329c2e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7217336Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1329c321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7218019Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1329c291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7218475Z frame #7: <unknown function> + 0x79d655 (0x7f133b115655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7218798Z frame #8: <unknown function> + 0xdc253 (0x7f12f2164253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7219010Z frame #9: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7219305Z frame #10: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7219309Z 
2025-07-17T11:07:26.7219400Z SIGABRT(6), PID: 1817866, Thread 1817913: 
2025-07-17T11:07:26.7219888Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7220097Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7220306Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7220505Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7221015Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7221220Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7221434Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7221438Z 
2025-07-17T11:07:26.7221526Z SIGABRT(6), PID: 1817873, Thread 1818373: 
2025-07-17T11:07:26.7221998Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7222204Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7222408Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7222605Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7223107Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7223313Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7223519Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7223523Z 
2025-07-17T11:07:26.7223612Z SIGABRT(6), PID: 1817869, Thread 1818094: 
2025-07-17T11:07:26.7224078Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7224287Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7224490Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7224691Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7225185Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7225396Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7225601Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7225605Z 
2025-07-17T11:07:26.7225695Z SIGABRT(6), PID: 1817871, Thread 1818194: 
2025-07-17T11:07:26.7226232Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7226504Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7226767Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7226972Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7227464Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7227748Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7227956Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7227960Z 
2025-07-17T11:07:26.7228050Z SIGABRT(6), PID: 1817866, Thread 1817915: 
2025-07-17T11:07:26.7228518Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7228727Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7228932Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7229130Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7229626Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7229837Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7230044Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7230049Z 
2025-07-17T11:07:26.7230142Z SIGABRT(6), PID: 1817873, Thread 1818374: 
2025-07-17T11:07:26.7230608Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7230817Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7231019Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7231214Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7231709Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7231915Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7232120Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7232132Z 
2025-07-17T11:07:26.7232223Z SIGABRT(6), PID: 1817871, Thread 1818197: 
2025-07-17T11:07:26.7232689Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7232898Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7233101Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7233300Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7233790Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7233999Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7234277Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7234344Z 
2025-07-17T11:07:26.7234436Z SIGABRT(6), PID: 1817869, Thread 1818096: 
2025-07-17T11:07:26.7234967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7235177Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7235442Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7235643Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7236137Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7236350Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7236562Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7236566Z 
2025-07-17T11:07:26.7236655Z SIGABRT(6), PID: 1817872, Thread 1822224: 
2025-07-17T11:07:26.7237127Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7237334Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7237539Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7237738Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7238118Z frame #4: <unknown function> + 0x5a02023 (0x7fdbd9fbb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7238689Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fdbd9fbf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7239293Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fdbd9fb61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7239676Z frame #7: <unknown function> + 0x79d655 (0x7fdbeb4a2655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7239910Z frame #8: <unknown function> + 0xdc253 (0x7fdba24f1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7240117Z frame #9: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7240345Z frame #10: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7240350Z 
2025-07-17T11:07:26.7240440Z SIGABRT(6), PID: 1817866, Thread 1817918: 
2025-07-17T11:07:26.7240914Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7241122Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7241326Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7241525Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7242027Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7242234Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7242528Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7242593Z 
2025-07-17T11:07:26.7242684Z SIGABRT(6), PID: 1817873, Thread 1818375: 
2025-07-17T11:07:26.7243232Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7243444Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7243706Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7243903Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7244399Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7244605Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7244816Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7244821Z 
2025-07-17T11:07:26.7244910Z SIGABRT(6), PID: 1817871, Thread 1818200: 
2025-07-17T11:07:26.7245376Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7245583Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7245787Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7245988Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7246481Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7246693Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7246901Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7246905Z 
2025-07-17T11:07:26.7246997Z SIGABRT(6), PID: 1817869, Thread 1818100: 
2025-07-17T11:07:26.7247457Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7247663Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7247865Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7248062Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7248560Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7248770Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7248976Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7248982Z 
2025-07-17T11:07:26.7249074Z SIGABRT(6), PID: 1817866, Thread 1817921: 
2025-07-17T11:07:26.7249538Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7249743Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7249946Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7250149Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7250721Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7250992Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7251257Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7251262Z 
2025-07-17T11:07:26.7251355Z SIGABRT(6), PID: 1817873, Thread 1818376: 
2025-07-17T11:07:26.7251881Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7252087Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7252288Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7252484Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7252981Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7253191Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7253396Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7253400Z 
2025-07-17T11:07:26.7253491Z SIGABRT(6), PID: 1817871, Thread 1818203: 
2025-07-17T11:07:26.7253960Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7254163Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7254370Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7254568Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7255063Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7255271Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7255484Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7255490Z 
2025-07-17T11:07:26.7255579Z SIGABRT(6), PID: 1817869, Thread 1818104: 
2025-07-17T11:07:26.7256115Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7256320Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7256529Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7256731Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7257228Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7257433Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7257643Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7257646Z 
2025-07-17T11:07:26.7257737Z SIGABRT(6), PID: 1817866, Thread 1817923: 
2025-07-17T11:07:26.7258208Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7258414Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7258706Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7258975Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7259547Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7259760Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7260048Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7260141Z SIGABRT(6), PID: 1817871, Thread 1818207: 
2025-07-17T11:07:26.7260612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7260817Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7261028Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7261223Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7261720Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7261929Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7262139Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7262143Z 
2025-07-17T11:07:26.7262146Z 
2025-07-17T11:07:26.7262234Z SIGABRT(6), PID: 1817873, Thread 1818377: 
2025-07-17T11:07:26.7262701Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7262905Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7263107Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7263301Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7263798Z frame #4: <unknown function> + 0x3475ab (0x7f495ef9f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7264003Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7264210Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7264299Z SIGABRT(6), PID: 1817869, Thread 1818107: 
2025-07-17T11:07:26.7264769Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7264977Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7265187Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7265383Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7265880Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7266090Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7266297Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7266302Z 
2025-07-17T11:07:26.7266305Z 
2025-07-17T11:07:26.7266398Z SIGABRT(6), PID: 1817873, Thread 1818396: 
2025-07-17T11:07:26.7266933Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7267501Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f48707fa08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7267711Z frame #2: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7267970Z frame #3: pthread_kill + 0x12c (0x7f49621099fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7268135Z frame #4: raise + 0x16 (0x7f49620b5476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7268294Z frame #5: abort + 0xd3 (0x7f496209b7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7268531Z frame #6: <unknown function> + 0x972cc (0x7f47de3832cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7268760Z frame #7: <unknown function> + 0x95f47 (0x7f47de381f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7268986Z frame #8: <unknown function> + 0x34b71 (0x7f47de320b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7269195Z frame #9: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7269408Z frame #10: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7269412Z 
2025-07-17T11:07:26.7269506Z SIGABRT(6), PID: 1817866, Thread 1817926: 
2025-07-17T11:07:26.7269981Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7270190Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7270393Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7270593Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7271091Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7271303Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7271513Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7271519Z 
2025-07-17T11:07:26.7271609Z SIGABRT(6), PID: 1817871, Thread 1818211: 
2025-07-17T11:07:26.7272081Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7272288Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7272492Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7272695Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7273189Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7273397Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7273608Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7273614Z 
2025-07-17T11:07:26.7273705Z SIGABRT(6), PID: 1817869, Thread 1818111: 
2025-07-17T11:07:26.7274167Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7274375Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7274645Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7274924Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7275476Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7275695Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7275965Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7275970Z 
2025-07-17T11:07:26.7276061Z SIGABRT(6), PID: 1817873, Thread 1821508: 
2025-07-17T11:07:26.7276525Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7276731Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7276898Z frame #2: __poll + 0x4f (0x7f496218bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7277309Z frame #3: <unknown function> + 0x489f (0x7f4961c9789f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.7277548Z frame #4: <unknown function> + 0x1483bc (0x55b7d73ec3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7277795Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7278025Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7278260Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7278488Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7278734Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55b7d73d4a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7278969Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7279215Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b7d73d2c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7279442Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7279680Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7279903Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7280138Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7280358Z frame #16: <unknown function> + 0x147734 (0x55b7d73eb734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7280581Z frame #17: <unknown function> + 0x21ca82 (0x55b7d74c0a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7280800Z frame #18: <unknown function> + 0x21c9e4 (0x55b7d74c09e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7281015Z frame #19: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7281230Z frame #20: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7281323Z SIGABRT(6), PID: 1817867, Thread 1822214: 
2025-07-17T11:07:26.7281800Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7282010Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7282217Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7282416Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7282863Z frame #4: <unknown function> + 0x5a02023 (0x7f1329c2e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7283544Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1329c321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7284147Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1329c291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7284589Z frame #7: <unknown function> + 0x79d655 (0x7f133b115655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7284819Z frame #8: <unknown function> + 0xdc253 (0x7f12f2164253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7285026Z frame #9: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7285245Z frame #10: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7285249Z 
2025-07-17T11:07:26.7285252Z 
2025-07-17T11:07:26.7285342Z SIGABRT(6), PID: 1817871, Thread 1818215: 
2025-07-17T11:07:26.7285822Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7286030Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7286235Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7286434Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7286938Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7287147Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7287358Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7287364Z 
2025-07-17T11:07:26.7287454Z SIGABRT(6), PID: 1817866, Thread 1817930: 
2025-07-17T11:07:26.7287926Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7288132Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7288339Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7288540Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7289044Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7289253Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7289467Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7289471Z 
2025-07-17T11:07:26.7289561Z SIGABRT(6), PID: 1817869, Thread 1818115: 
2025-07-17T11:07:26.7290036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7290242Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7290447Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7290641Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7291211Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7291602Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7291874Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7291878Z 
2025-07-17T11:07:26.7291976Z SIGABRT(6), PID: 1817871, Thread 1818219: 
2025-07-17T11:07:26.7292503Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7292711Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7292914Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7293116Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7293609Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7293820Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7294025Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7294031Z 
2025-07-17T11:07:26.7294123Z SIGABRT(6), PID: 1817866, Thread 1817934: 
2025-07-17T11:07:26.7294589Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7294797Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7295000Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7295199Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7295696Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7295969Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7296178Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7296184Z 
2025-07-17T11:07:26.7296273Z SIGABRT(6), PID: 1817869, Thread 1818119: 
2025-07-17T11:07:26.7296737Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7296947Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7297152Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7297354Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7297848Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7298059Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7298267Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7298271Z 
2025-07-17T11:07:26.7298360Z SIGABRT(6), PID: 1817871, Thread 1818223: 
2025-07-17T11:07:26.7298822Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7299032Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7299418Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7299628Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7300202Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7300489Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7300699Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7300703Z 
2025-07-17T11:07:26.7300798Z SIGABRT(6), PID: 1817872, Thread 1822225: 
2025-07-17T11:07:26.7301266Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7301477Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7301686Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7301892Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7302269Z frame #4: <unknown function> + 0x5a02023 (0x7fdbd9fbb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7302841Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fdbd9fbf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7303436Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fdbd9fb61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7303819Z frame #7: <unknown function> + 0x79d655 (0x7fdbeb4a2655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7304056Z frame #8: <unknown function> + 0xdc253 (0x7fdba24f1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7304263Z frame #9: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7304479Z frame #10: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7304485Z 
2025-07-17T11:07:26.7304573Z SIGABRT(6), PID: 1817866, Thread 1817938: 
2025-07-17T11:07:26.7305048Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7305251Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7305456Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7305658Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7306162Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7306367Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7306578Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7306582Z 
2025-07-17T11:07:26.7306672Z SIGABRT(6), PID: 1817869, Thread 1818123: 
2025-07-17T11:07:26.7307149Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7307353Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7307655Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7307915Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7308476Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7308688Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7308966Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7308970Z 
2025-07-17T11:07:26.7309059Z SIGABRT(6), PID: 1817871, Thread 1818227: 
2025-07-17T11:07:26.7309531Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7309739Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7309948Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7310146Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7310648Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7310853Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7311063Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7311066Z 
2025-07-17T11:07:26.7311153Z SIGABRT(6), PID: 1817873, Thread 1821842: 
2025-07-17T11:07:26.7311622Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7311829Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7312032Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7312261Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f49621072dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7312702Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f489c542048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7313123Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f489c544bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7313352Z frame #6: <unknown function> + 0xdc253 (0x7f486ea66253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7313561Z frame #7: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7313770Z frame #8: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7313776Z 
2025-07-17T11:07:26.7313865Z SIGABRT(6), PID: 1817869, Thread 1818127: 
2025-07-17T11:07:26.7314328Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7314539Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7314742Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7314943Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7315440Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7315719Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7315984Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7316078Z SIGABRT(6), PID: 1817866, Thread 1817942: 
2025-07-17T11:07:26.7316601Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7316815Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7317081Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7317284Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7317775Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7317989Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7318195Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7318199Z 
2025-07-17T11:07:26.7318203Z 
2025-07-17T11:07:26.7318294Z SIGABRT(6), PID: 1817871, Thread 1818231: 
2025-07-17T11:07:26.7318759Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7318969Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7319176Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7319376Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7319869Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7320080Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7320288Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7320291Z 
2025-07-17T11:07:26.7320386Z SIGABRT(6), PID: 1817869, Thread 1818131: 
2025-07-17T11:07:26.7320849Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7321059Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7321261Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7321461Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7321960Z frame #4: <unknown function> + 0x3475ab (0x7ffa1ed2f5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7322172Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7322380Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7322386Z 
2025-07-17T11:07:26.7322475Z SIGABRT(6), PID: 1817871, Thread 1818234: 
2025-07-17T11:07:26.7322951Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7323156Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7323360Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7323556Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7324190Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7324462Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7324676Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7324827Z SIGABRT(6), PID: 1817866, Thread 1817946: 
2025-07-17T11:07:26.7325299Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7325504Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7325711Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7325913Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7326413Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7326622Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7326832Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7326838Z 
2025-07-17T11:07:26.7326925Z SIGABRT(6), PID: 1817873, Thread 1821843: 
2025-07-17T11:07:26.7327399Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7327601Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7327808Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7328029Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f49621072dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7328505Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f489c53c741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7328735Z frame #5: <unknown function> + 0xdc253 (0x7f486ea66253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7328945Z frame #6: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7329154Z frame #7: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7329158Z 
2025-07-17T11:07:26.7329161Z 
2025-07-17T11:07:26.7334307Z SIGABRT(6), PID: 1817869, Thread 1818397: 
2025-07-17T11:07:26.7334861Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7335347Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7ff93058f08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7335570Z frame #2: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7335774Z frame #3: pthread_kill + 0x12c (0x7ffa21e999fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7336054Z frame #4: raise + 0x16 (0x7ffa21e45476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7336232Z frame #5: abort + 0xd3 (0x7ffa21e2b7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7336476Z frame #6: <unknown function> + 0x972cc (0x7ff89e1182cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7336707Z frame #7: <unknown function> + 0x95f47 (0x7ff89e116f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7336937Z frame #8: <unknown function> + 0x34b71 (0x7ff89e0b5b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7337289Z frame #9: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7337593Z frame #10: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7337600Z 
2025-07-17T11:07:26.7337773Z SIGABRT(6), PID: 1817873, Thread 1821844: 
2025-07-17T11:07:26.7338271Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7338590Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7338756Z frame #2: ioctl + 0x3f (0x7f496218d94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7339006Z frame #3: <unknown function> + 0x13eef0 (0x7f47de42aef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7339239Z frame #4: <unknown function> + 0x1378c4 (0x7f47de4238c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7339474Z frame #5: <unknown function> + 0x95b2b (0x7f47de381b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7339703Z frame #6: <unknown function> + 0x34b71 (0x7f47de320b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7339918Z frame #7: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7340131Z frame #8: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7340137Z 
2025-07-17T11:07:26.7340234Z SIGABRT(6), PID: 1817871, Thread 1818238: 
2025-07-17T11:07:26.7340717Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7340928Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7341131Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7341343Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7341854Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7342064Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7342272Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7343586Z 
2025-07-17T11:07:26.7343720Z SIGABRT(6), PID: 1817866, Thread 1817950: 
2025-07-17T11:07:26.7344237Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7344469Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7344679Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7344907Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7345433Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7345651Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7345882Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7345886Z 
2025-07-17T11:07:26.7345985Z SIGABRT(6), PID: 1817867, Thread 1822215: 
2025-07-17T11:07:26.7346475Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7346682Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7346992Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7347197Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7347667Z frame #4: <unknown function> + 0x5a02023 (0x7f1329c2e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7348257Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1329c321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7348922Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1329c291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7349316Z frame #7: <unknown function> + 0x79d655 (0x7f133b115655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7349555Z frame #8: <unknown function> + 0xdc253 (0x7f12f2164253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7349768Z frame #9: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7349984Z frame #10: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7349988Z 
2025-07-17T11:07:26.7350085Z SIGABRT(6), PID: 1817869, Thread 1821309: 
2025-07-17T11:07:26.7350566Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7350777Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7350949Z frame #2: __poll + 0x4f (0x7ffa21f1bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7351361Z frame #3: <unknown function> + 0x489f (0x7ffa21a2789f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.7351599Z frame #4: <unknown function> + 0x1483bc (0x55f859e893bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7351852Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7352079Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7352420Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7352657Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7352904Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55f859e71a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7353133Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7353385Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f859e6fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7353609Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7353847Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7354069Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7354307Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7354526Z frame #16: <unknown function> + 0x147734 (0x55f859e88734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7354746Z frame #17: <unknown function> + 0x21ca82 (0x55f859f5da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7354960Z frame #18: <unknown function> + 0x21c9e4 (0x55f859f5d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7355181Z frame #19: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7355469Z frame #20: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7355473Z 
2025-07-17T11:07:26.7355570Z SIGABRT(6), PID: 1817873, Thread 1822145: 
2025-07-17T11:07:26.7356109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7356383Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7356547Z frame #2: __poll + 0x4f (0x7f496218bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7356765Z frame #3: <unknown function> + 0xd4540 (0x7f482659a540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7356969Z frame #4: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7357178Z frame #5: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7357185Z 
2025-07-17T11:07:26.7357277Z SIGABRT(6), PID: 1817871, Thread 1818242: 
2025-07-17T11:07:26.7357758Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7357967Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7358178Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7358383Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7358895Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7359105Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7359316Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7359322Z 
2025-07-17T11:07:26.7359410Z SIGABRT(6), PID: 1817866, Thread 1817956: 
2025-07-17T11:07:26.7359883Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7360089Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7360376Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7360582Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7361087Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7361291Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7361502Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7361506Z 
2025-07-17T11:07:26.7361597Z SIGABRT(6), PID: 1817873, Thread 1822154: 
2025-07-17T11:07:26.7362061Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7362271Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7362434Z frame #2: __poll + 0x4f (0x7f496218bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7362646Z frame #3: <unknown function> + 0xd65a2 (0x7f482659c5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7362856Z frame #4: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7363061Z frame #5: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7363136Z 
2025-07-17T11:07:26.7363227Z SIGABRT(6), PID: 1817871, Thread 1818246: 
2025-07-17T11:07:26.7363763Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7363974Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7364241Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7364440Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7364939Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7365148Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7365359Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7365366Z 
2025-07-17T11:07:26.7365453Z SIGABRT(6), PID: 1817866, Thread 1817959: 
2025-07-17T11:07:26.7365921Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7366126Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7366335Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7366529Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7367023Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7367227Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7367437Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7367441Z 
2025-07-17T11:07:26.7367530Z SIGABRT(6), PID: 1817873, Thread 1822158: 
2025-07-17T11:07:26.7367992Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7368263Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7368471Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7368664Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7368878Z frame #4: <unknown function> + 0xd1e7f (0x7f4826597e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7369079Z frame #5: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7369286Z frame #6: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7369290Z 
2025-07-17T11:07:26.7369380Z SIGABRT(6), PID: 1817871, Thread 1818250: 
2025-07-17T11:07:26.7369853Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7370059Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7370263Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7370458Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7370952Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7371229Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7371441Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7371615Z SIGABRT(6), PID: 1817872, Thread 1822226: 
2025-07-17T11:07:26.7372088Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7372357Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7372564Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7372765Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7373148Z frame #4: <unknown function> + 0x5a02023 (0x7fdbd9fbb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7373719Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fdbd9fbf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7374320Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fdbd9fb61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7374714Z frame #7: <unknown function> + 0x79d655 (0x7fdbeb4a2655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7374948Z frame #8: <unknown function> + 0xdc253 (0x7fdba24f1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7375159Z frame #9: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7375374Z frame #10: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7375379Z 
2025-07-17T11:07:26.7375383Z 
2025-07-17T11:07:26.7375475Z SIGABRT(6), PID: 1817869, Thread 1821908: 
2025-07-17T11:07:26.7376077Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7376291Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7376582Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7376811Z frame #3: pthread_cond_clockwait + 0x1fd (0x7ffa21e972dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7377273Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7ff95c2d7048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7377707Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7ff95c2d9bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7377946Z frame #6: <unknown function> + 0xdc253 (0x7ff92e7fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7378161Z frame #7: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7378373Z frame #8: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7378380Z 
2025-07-17T11:07:26.7378474Z SIGABRT(6), PID: 1817866, Thread 1817963: 
2025-07-17T11:07:26.7378952Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7379164Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7379368Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7379571Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7380164Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7380452Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7380670Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7380747Z 
2025-07-17T11:07:26.7380846Z SIGABRT(6), PID: 1817873, Thread 1822166: 
2025-07-17T11:07:26.7381318Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7381525Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7381726Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7381935Z frame #3: <unknown function> + 0x9ca53 (0x7f496210fa53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7382190Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55b7d73b8bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7382423Z frame #5: <unknown function> + 0x1d314f (0x55b7d747714f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7382646Z frame #6: <unknown function> + 0x1d2f6b (0x55b7d7476f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7382873Z frame #7: <unknown function> + 0x6df36 (0x55b7d7311f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7383116Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7383345Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7383586Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7383819Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7384060Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7384289Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7384523Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7384838Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7385078Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7385300Z frame #17: <unknown function> + 0x147734 (0x55b7d73eb734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7385518Z frame #18: <unknown function> + 0x21ca82 (0x55b7d74c0a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7385737Z frame #19: <unknown function> + 0x21c9e4 (0x55b7d74c09e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7385951Z frame #20: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7386164Z frame #21: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7386169Z 
2025-07-17T11:07:26.7386258Z SIGABRT(6), PID: 1817871, Thread 1818254: 
2025-07-17T11:07:26.7386735Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7386948Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7387157Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7387360Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7387871Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7388209Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7388430Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7388435Z 
2025-07-17T11:07:26.7388586Z SIGABRT(6), PID: 1817866, Thread 1817967: 
2025-07-17T11:07:26.7389062Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7389272Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7390797Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7391028Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7391544Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7391753Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7391968Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7391974Z 
2025-07-17T11:07:26.7392066Z SIGABRT(6), PID: 1817871, Thread 1818258: 
2025-07-17T11:07:26.7392545Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7392755Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7392968Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7393172Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7393670Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7393878Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7394179Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7394184Z 
2025-07-17T11:07:26.7394278Z SIGABRT(6), PID: 1817873, Thread 1822177: 
2025-07-17T11:07:26.7394747Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7394954Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7395121Z frame #2: read + 0x4c (0x7f496218781c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7395307Z frame #3: _Py_read + 0x4c (0x55b7d747ea6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7395540Z frame #4: <unknown function> + 0x208c3b (0x55b7d74acc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7395764Z frame #5: <unknown function> + 0x14648d (0x55b7d73ea48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7396014Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55b7d73f9215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7396237Z frame #7: <unknown function> + 0x208ae7 (0x55b7d74acae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7396457Z frame #8: <unknown function> + 0x208472 (0x55b7d74ac472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7396675Z frame #9: <unknown function> + 0x1da6ee (0x55b7d747e6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7396904Z frame #10: <unknown function> + 0x1483bc (0x55b7d73ec3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7397219Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7397446Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7397684Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55b7d73d03bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7397906Z frame #14: <unknown function> + 0x147734 (0x55b7d73eb734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7398221Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b7d73d2c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7398446Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7398678Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7398999Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55b7d73e00ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7399240Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55b7d73d07b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7399467Z frame #20: <unknown function> + 0x147734 (0x55b7d73eb734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7399691Z frame #21: <unknown function> + 0x21ca82 (0x55b7d74c0a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7399910Z frame #22: <unknown function> + 0x21c9e4 (0x55b7d74c09e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7400123Z frame #23: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7400336Z frame #24: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7400340Z 
2025-07-17T11:07:26.7400434Z SIGABRT(6), PID: 1817869, Thread 1821909: 
2025-07-17T11:07:26.7400916Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7401133Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7401341Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7401565Z frame #3: pthread_cond_clockwait + 0x1fd (0x7ffa21e972dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7402137Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7ff95c2d1741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7402379Z frame #5: <unknown function> + 0xdc253 (0x7ff92e7fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7402593Z frame #6: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7402810Z frame #7: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7402814Z 
2025-07-17T11:07:26.7402905Z SIGABRT(6), PID: 1817866, Thread 1817970: 
2025-07-17T11:07:26.7403379Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7403586Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7403794Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7403999Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7404512Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7404721Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7404934Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7405003Z 
2025-07-17T11:07:26.7405095Z SIGABRT(6), PID: 1817871, Thread 1818263: 
2025-07-17T11:07:26.7405568Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7405774Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7406047Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7406254Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7406752Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7407027Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7407242Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7407248Z 
2025-07-17T11:07:26.7407337Z SIGABRT(6), PID: 1817869, Thread 1821910: 
2025-07-17T11:07:26.7407808Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7408014Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7408189Z frame #2: ioctl + 0x3f (0x7ffa21f1d94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7408427Z frame #3: <unknown function> + 0x13eef0 (0x7ff89e1bfef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7408660Z frame #4: <unknown function> + 0x1378c4 (0x7ff89e1b88c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7408891Z frame #5: <unknown function> + 0x95b2b (0x7ff89e116b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7409118Z frame #6: <unknown function> + 0x34b71 (0x7ff89e0b5b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7409329Z frame #7: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7409542Z frame #8: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7409546Z 
2025-07-17T11:07:26.7409633Z SIGABRT(6), PID: 1817866, Thread 1817974: 
2025-07-17T11:07:26.7410176Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7410389Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7410596Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7410798Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7411300Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7411509Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7411720Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7411724Z 
2025-07-17T11:07:26.7411818Z SIGABRT(6), PID: 1817871, Thread 1818267: 
2025-07-17T11:07:26.7412298Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7412503Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7412710Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7412910Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7413477Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7413685Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7413895Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7413962Z 
2025-07-17T11:07:26.7414058Z SIGABRT(6), PID: 1817867, Thread 1822216: 
2025-07-17T11:07:26.7414533Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7414741Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7415007Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7415212Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7415592Z frame #4: <unknown function> + 0x5a02023 (0x7f1329c2e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7416253Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1329c321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7416851Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1329c291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7417238Z frame #7: <unknown function> + 0x79d655 (0x7f133b115655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7417469Z frame #8: <unknown function> + 0xdc253 (0x7f12f2164253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7417680Z frame #9: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7417893Z frame #10: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7417897Z 
2025-07-17T11:07:26.7417988Z SIGABRT(6), PID: 1817869, Thread 1822141: 
2025-07-17T11:07:26.7418541Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7418760Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7418931Z frame #2: __poll + 0x4f (0x7ffa21f1bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7419147Z frame #3: <unknown function> + 0xd4540 (0x7ff8e632f540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7419354Z frame #4: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7419564Z frame #5: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7419568Z 
2025-07-17T11:07:26.7419656Z SIGABRT(6), PID: 1817866, Thread 1817978: 
2025-07-17T11:07:26.7420129Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7420339Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7420547Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7420747Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7421251Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7421457Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7421748Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7421752Z 
2025-07-17T11:07:26.7421842Z SIGABRT(6), PID: 1817869, Thread 1822149: 
2025-07-17T11:07:26.7422316Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7422603Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7422771Z frame #2: __poll + 0x4f (0x7ffa21f1bbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7422986Z frame #3: <unknown function> + 0xd65a2 (0x7ff8e63315a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7423389Z frame #4: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7423601Z frame #5: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7423607Z 
2025-07-17T11:07:26.7423697Z SIGABRT(6), PID: 1817871, Thread 1818271: 
2025-07-17T11:07:26.7424172Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7424385Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7424592Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7424800Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7425300Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7425509Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7425721Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7425725Z 
2025-07-17T11:07:26.7425813Z SIGABRT(6), PID: 1817869, Thread 1822160: 
2025-07-17T11:07:26.7426283Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7426556Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7426767Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7426978Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7427204Z frame #4: <unknown function> + 0xd1e7f (0x7ff8e632ce7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7427418Z frame #5: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7427630Z frame #6: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7427725Z SIGABRT(6), PID: 1817866, Thread 1817983: 
2025-07-17T11:07:26.7428206Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7428416Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7428623Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7428826Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7429333Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7429544Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7429826Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7429830Z 
2025-07-17T11:07:26.7429834Z 
2025-07-17T11:07:26.7429929Z SIGABRT(6), PID: 1817872, Thread 1822227: 
2025-07-17T11:07:26.7430401Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fdba4284b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7430679Z frame #1: <unknown function> + 0x42520 (0x7fdc95b3c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7430885Z frame #2: <unknown function> + 0x91117 (0x7fdc95b8b117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7431086Z frame #3: pthread_cond_wait + 0x211 (0x7fdc95b8da41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7431553Z frame #4: <unknown function> + 0x5a02023 (0x7fdbd9fbb023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7432132Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fdbd9fbf1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7432730Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fdbd9fb61d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7433121Z frame #7: <unknown function> + 0x79d655 (0x7fdbeb4a2655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7433355Z frame #8: <unknown function> + 0xdc253 (0x7fdba24f1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7433568Z frame #9: <unknown function> + 0x94ac3 (0x7fdc95b8eac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7433786Z frame #10: <unknown function> + 0x126850 (0x7fdc95c20850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7433879Z SIGABRT(6), PID: 1817871, Thread 1818274: 
2025-07-17T11:07:26.7434350Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7434561Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7434768Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7435035Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7435548Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7435761Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7435971Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7435977Z 
2025-07-17T11:07:26.7435980Z 
2025-07-17T11:07:26.7436073Z SIGABRT(6), PID: 1817869, Thread 1822165: 
2025-07-17T11:07:26.7436545Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7436751Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7436959Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7437165Z frame #3: <unknown function> + 0x9ca53 (0x7ffa21e9fa53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7437423Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55f859e55bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7437654Z frame #5: <unknown function> + 0x1d314f (0x55f859f1414f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7437879Z frame #6: <unknown function> + 0x1d2f6b (0x55f859f13f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7438170Z frame #7: <unknown function> + 0x6df36 (0x55f859daef36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7438413Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7438642Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7438949Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7439176Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7439412Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7439702Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7439942Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7440169Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7440401Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7440622Z frame #17: <unknown function> + 0x147734 (0x55f859e88734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7440843Z frame #18: <unknown function> + 0x21ca82 (0x55f859f5da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7441066Z frame #19: <unknown function> + 0x21c9e4 (0x55f859f5d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7441278Z frame #20: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7441491Z frame #21: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7441497Z 
2025-07-17T11:07:26.7441590Z SIGABRT(6), PID: 1817866, Thread 1817987: 
2025-07-17T11:07:26.7442070Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7442283Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7442493Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7442760Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7443269Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7443480Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7443693Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7443699Z 
2025-07-17T11:07:26.7443791Z SIGABRT(6), PID: 1817871, Thread 1818278: 
2025-07-17T11:07:26.7444259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7444472Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7444677Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7444880Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7445378Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7445591Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7445800Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7445870Z 
2025-07-17T11:07:26.7445964Z SIGABRT(6), PID: 1817866, Thread 1817992: 
2025-07-17T11:07:26.7446436Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7446647Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7446918Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7447122Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7447616Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7447884Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7448098Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7448102Z 
2025-07-17T11:07:26.7448193Z SIGABRT(6), PID: 1817871, Thread 1818282: 
2025-07-17T11:07:26.7448666Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7448877Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7449079Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7449278Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7449770Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7449985Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7450193Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7450197Z 
2025-07-17T11:07:26.7450287Z SIGABRT(6), PID: 1817869, Thread 1822175: 
2025-07-17T11:07:26.7450757Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7451033Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7451203Z frame #2: read + 0x4c (0x7ffa21f1781c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7451390Z frame #3: _Py_read + 0x4c (0x55f859f1ba6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7451623Z frame #4: <unknown function> + 0x208c3b (0x55f859f49c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7451847Z frame #5: <unknown function> + 0x14648d (0x55f859e8748d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7452094Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55f859e96215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7452319Z frame #7: <unknown function> + 0x208ae7 (0x55f859f49ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7452535Z frame #8: <unknown function> + 0x208472 (0x55f859f49472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7452759Z frame #9: <unknown function> + 0x1da6ee (0x55f859f1b6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7452981Z frame #10: <unknown function> + 0x1483bc (0x55f859e893bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7453222Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7453449Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7453687Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55f859e6d3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7453977Z frame #14: <unknown function> + 0x147734 (0x55f859e88734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7454230Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f859e6fc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7454454Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7454753Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7454975Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55f859e7d0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7455210Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55f859e6d7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7455487Z frame #20: <unknown function> + 0x147734 (0x55f859e88734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7455711Z frame #21: <unknown function> + 0x21ca82 (0x55f859f5da82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7456009Z frame #22: <unknown function> + 0x21c9e4 (0x55f859f5d9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7456230Z frame #23: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7456444Z frame #24: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7456450Z 
2025-07-17T11:07:26.7456547Z SIGABRT(6), PID: 1817873, Thread 1822204: 
2025-07-17T11:07:26.7457021Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7457234Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7457442Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7457640Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7458018Z frame #4: <unknown function> + 0x5a02023 (0x7f48a6530023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7458585Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f48a65341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7459265Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f48a652b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7459654Z frame #7: <unknown function> + 0x79d655 (0x7f48b7a1b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7459889Z frame #8: <unknown function> + 0xdc253 (0x7f486ea66253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7460095Z frame #9: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7460310Z frame #10: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7460314Z 
2025-07-17T11:07:26.7460403Z SIGABRT(6), PID: 1817871, Thread 1818287: 
2025-07-17T11:07:26.7460886Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7461096Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7461306Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7461507Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7462013Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7462318Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7462533Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7462537Z 
2025-07-17T11:07:26.7462625Z SIGABRT(6), PID: 1817866, Thread 1817998: 
2025-07-17T11:07:26.7463109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7463433Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7463641Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7463840Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7464407Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7464623Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7464832Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7464836Z 
2025-07-17T11:07:26.7464925Z SIGABRT(6), PID: 1817871, Thread 1818291: 
2025-07-17T11:07:26.7465404Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7465610Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7465818Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7466022Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7466523Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7466731Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7466943Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7466948Z 
2025-07-17T11:07:26.7467037Z SIGABRT(6), PID: 1817867, Thread 1822217: 
2025-07-17T11:07:26.7467576Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7467785Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7467989Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7468185Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7468566Z frame #4: <unknown function> + 0x5a02023 (0x7f1329c2e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7469134Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1329c321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7469725Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1329c291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7470110Z frame #7: <unknown function> + 0x79d655 (0x7f133b115655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7470344Z frame #8: <unknown function> + 0xdc253 (0x7f12f2164253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7470621Z frame #9: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7470842Z frame #10: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7470935Z SIGABRT(6), PID: 1817866, Thread 1818002: 
2025-07-17T11:07:26.7471406Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7471676Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7471880Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7472080Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7472637Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7472856Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7473064Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7473068Z 
2025-07-17T11:07:26.7473071Z 
2025-07-17T11:07:26.7473163Z SIGABRT(6), PID: 1817871, Thread 1818295: 
2025-07-17T11:07:26.7473637Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7473845Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7474049Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7474252Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7474751Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7474966Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7475174Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7475178Z 
2025-07-17T11:07:26.7475270Z SIGABRT(6), PID: 1817866, Thread 1818006: 
2025-07-17T11:07:26.7475802Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7476013Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7476213Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7476414Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7476912Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7477125Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7477333Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7477342Z 
2025-07-17T11:07:26.7477432Z SIGABRT(6), PID: 1817871, Thread 1818299: 
2025-07-17T11:07:26.7477905Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7478111Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7478316Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7478580Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7479079Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7479288Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7479561Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7479567Z 
2025-07-17T11:07:26.7479657Z SIGABRT(6), PID: 1817866, Thread 1818342: 
2025-07-17T11:07:26.7480127Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7480393Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7480605Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7480805Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7481302Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7481509Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7481724Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7481728Z 
2025-07-17T11:07:26.7481817Z SIGABRT(6), PID: 1817871, Thread 1818303: 
2025-07-17T11:07:26.7482293Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7482501Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7482710Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7482908Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7483402Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7483676Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7483891Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7483896Z 
2025-07-17T11:07:26.7483984Z SIGABRT(6), PID: 1817866, Thread 1818343: 
2025-07-17T11:07:26.7484459Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7484663Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7484871Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7485071Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7485565Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7485774Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7485984Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7485988Z 
2025-07-17T11:07:26.7486076Z SIGABRT(6), PID: 1817871, Thread 1818306: 
2025-07-17T11:07:26.7486547Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7486822Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7487030Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7487228Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7487729Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7488001Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7488211Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7488215Z 
2025-07-17T11:07:26.7488365Z SIGABRT(6), PID: 1817866, Thread 1818344: 
2025-07-17T11:07:26.7488836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7489046Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7489249Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7489449Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7489946Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7490154Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7490361Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7490367Z 
2025-07-17T11:07:26.7490457Z SIGABRT(6), PID: 1817871, Thread 1818309: 
2025-07-17T11:07:26.7490929Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7491139Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7491344Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7491612Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7492108Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7492319Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7492529Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7492535Z 
2025-07-17T11:07:26.7492626Z SIGABRT(6), PID: 1817866, Thread 1818345: 
2025-07-17T11:07:26.7493092Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7493301Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7493503Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7493708Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7494202Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7494415Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7494620Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7494710Z 
2025-07-17T11:07:26.7494804Z SIGABRT(6), PID: 1817869, Thread 1822196: 
2025-07-17T11:07:26.7495274Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7495482Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7495754Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7496029Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7496406Z frame #4: <unknown function> + 0x5a02023 (0x7ff9662c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7497061Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9662c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7497659Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9662c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7498045Z frame #7: <unknown function> + 0x79d655 (0x7ff9777a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7498280Z frame #8: <unknown function> + 0xdc253 (0x7ff92e7fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7498493Z frame #9: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7498705Z frame #10: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7498713Z 
2025-07-17T11:07:26.7498805Z SIGABRT(6), PID: 1817871, Thread 1818312: 
2025-07-17T11:07:26.7499279Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7499490Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7499700Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7499898Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7500475Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7500687Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7500902Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7500906Z 
2025-07-17T11:07:26.7500996Z SIGABRT(6), PID: 1817873, Thread 1822205: 
2025-07-17T11:07:26.7501470Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7501675Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7501877Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7502075Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7502451Z frame #4: <unknown function> + 0x5a02023 (0x7f48a6530023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7503011Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f48a65341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7503605Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f48a652b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7504067Z frame #7: <unknown function> + 0x79d655 (0x7f48b7a1b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7504300Z frame #8: <unknown function> + 0xdc253 (0x7f486ea66253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7504582Z frame #9: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7504799Z frame #10: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7504803Z 
2025-07-17T11:07:26.7504892Z SIGABRT(6), PID: 1817866, Thread 1818346: 
2025-07-17T11:07:26.7505433Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7505646Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7505853Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7506051Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7506554Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7506762Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7506973Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7506977Z 
2025-07-17T11:07:26.7507067Z SIGABRT(6), PID: 1817867, Thread 1822218: 
2025-07-17T11:07:26.7507539Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7507744Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7507949Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7508144Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7508583Z frame #4: <unknown function> + 0x5a02023 (0x7f1329c2e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7509151Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1329c321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7509741Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1329c291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7510122Z frame #7: <unknown function> + 0x79d655 (0x7f133b115655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7510349Z frame #8: <unknown function> + 0xdc253 (0x7f12f2164253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7510559Z frame #9: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7510772Z frame #10: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7510777Z 
2025-07-17T11:07:26.7510868Z SIGABRT(6), PID: 1817871, Thread 1818315: 
2025-07-17T11:07:26.7511343Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7511554Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7511828Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7512029Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7512528Z frame #4: <unknown function> + 0x3475ab (0x7fa3c795d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7512807Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7513015Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7513020Z 
2025-07-17T11:07:26.7513110Z SIGABRT(6), PID: 1817866, Thread 1818347: 
2025-07-17T11:07:26.7513638Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7513850Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7514058Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7514259Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7514754Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7514970Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7515179Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7515183Z 
2025-07-17T11:07:26.7515274Z SIGABRT(6), PID: 1817871, Thread 1818390: 
2025-07-17T11:07:26.7515745Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7516204Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fa2d91be08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7516411Z frame #2: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7516606Z frame #3: pthread_kill + 0x12c (0x7fa3caac79fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7516840Z frame #4: raise + 0x16 (0x7fa3caa73476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7517008Z frame #5: abort + 0xd3 (0x7fa3caa597f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7517245Z frame #6: <unknown function> + 0x972cc (0x7fa246d472cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7517477Z frame #7: <unknown function> + 0x95f47 (0x7fa246d45f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7517704Z frame #8: <unknown function> + 0x34b71 (0x7fa246ce4b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7517920Z frame #9: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7518133Z frame #10: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7518138Z 
2025-07-17T11:07:26.7518231Z SIGABRT(6), PID: 1817866, Thread 1818348: 
2025-07-17T11:07:26.7518706Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7518916Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7519119Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7519319Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7519823Z frame #4: <unknown function> + 0x3475ab (0x7fee1d3885ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7520100Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7520310Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7520317Z 
2025-07-17T11:07:26.7520406Z SIGABRT(6), PID: 1817871, Thread 1820714: 
2025-07-17T11:07:26.7520956Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7521163Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7521335Z frame #2: __poll + 0x4f (0x7fa3cab49bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7521805Z frame #3: <unknown function> + 0x489f (0x7fa3ca65589f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.7522042Z frame #4: <unknown function> + 0x1483bc (0x556e5a9753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7522286Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7522515Z frame #6: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7522751Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7522979Z frame #8: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7523221Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x556e5a95da65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7523452Z frame #10: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7523702Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x556e5a95bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7523930Z frame #12: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7524167Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7524390Z frame #14: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7524623Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7524930Z frame #16: <unknown function> + 0x147734 (0x556e5a974734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7525151Z frame #17: <unknown function> + 0x21ca82 (0x556e5aa49a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7525373Z frame #18: <unknown function> + 0x21c9e4 (0x556e5aa499e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7525590Z frame #19: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7525811Z frame #20: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7525901Z SIGABRT(6), PID: 1817866, Thread 1818391: 
2025-07-17T11:07:26.7526380Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7526835Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fed2ebe608a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7527051Z frame #2: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7527245Z frame #3: pthread_kill + 0x12c (0x7fee204f29fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7527409Z frame #4: raise + 0x16 (0x7fee2049e476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7527567Z frame #5: abort + 0xd3 (0x7fee204847f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7527809Z frame #6: <unknown function> + 0x972cc (0x7fec9c76f2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7528112Z frame #7: <unknown function> + 0x95f47 (0x7fec9c76df47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7528342Z frame #8: <unknown function> + 0x34b71 (0x7fec9c70cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7528554Z frame #9: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7528834Z frame #10: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7528838Z 
2025-07-17T11:07:26.7528842Z 
2025-07-17T11:07:26.7528931Z SIGABRT(6), PID: 1817866, Thread 1821707: 
2025-07-17T11:07:26.7529406Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7529676Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7529848Z frame #2: __poll + 0x4f (0x7fee20574bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7530254Z frame #3: <unknown function> + 0x489f (0x7fee2008089f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.7530480Z frame #4: <unknown function> + 0x1483bc (0x5622549ca3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7530722Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7530952Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7531186Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7531410Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7531652Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5622549b2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7531883Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7532129Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5622549b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7532355Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7532592Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7532884Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7533120Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7533341Z frame #16: <unknown function> + 0x147734 (0x5622549c9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7533561Z frame #17: <unknown function> + 0x21ca82 (0x562254a9ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7533780Z frame #18: <unknown function> + 0x21c9e4 (0x562254a9e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7533994Z frame #19: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7534208Z frame #20: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7534212Z 
2025-07-17T11:07:26.7534301Z SIGABRT(6), PID: 1817869, Thread 1822197: 
2025-07-17T11:07:26.7534781Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7534994Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7535201Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7535403Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7535846Z frame #4: <unknown function> + 0x5a02023 (0x7ff9662c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7536489Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9662c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7537163Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9662c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7537545Z frame #7: <unknown function> + 0x79d655 (0x7ff9777a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7537851Z frame #8: <unknown function> + 0xdc253 (0x7ff92e7fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7538066Z frame #9: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7538289Z frame #10: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7538380Z SIGABRT(6), PID: 1817871, Thread 1822050: 
2025-07-17T11:07:26.7538861Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7539072Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7539282Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7539506Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa3caac52dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7539955Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fa304f06048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7540375Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fa304f08bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7540608Z frame #6: <unknown function> + 0xdc253 (0x7fa2d742a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7540818Z frame #7: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7541040Z frame #8: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7541127Z 
2025-07-17T11:07:26.7541131Z 
2025-07-17T11:07:26.7541227Z SIGABRT(6), PID: 1817873, Thread 1822206: 
2025-07-17T11:07:26.7541697Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7541904Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7542108Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7542310Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7542682Z frame #4: <unknown function> + 0x5a02023 (0x7f48a6530023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7543250Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f48a65341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7543840Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f48a652b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7544227Z frame #7: <unknown function> + 0x79d655 (0x7f48b7a1b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7544453Z frame #8: <unknown function> + 0xdc253 (0x7f486ea66253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7544762Z frame #9: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7544972Z frame #10: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7544976Z 
2025-07-17T11:07:26.7545067Z SIGABRT(6), PID: 1817867, Thread 1822219: 
2025-07-17T11:07:26.7545606Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f12f3ef7b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7545817Z frame #1: <unknown function> + 0x42520 (0x7f13e57af520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7546020Z frame #2: <unknown function> + 0x91117 (0x7f13e57fe117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7546283Z frame #3: pthread_cond_wait + 0x211 (0x7f13e5800a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7546657Z frame #4: <unknown function> + 0x5a02023 (0x7f1329c2e023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7547224Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1329c321ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7547814Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1329c291d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7548196Z frame #7: <unknown function> + 0x79d655 (0x7f133b115655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7548424Z frame #8: <unknown function> + 0xdc253 (0x7f12f2164253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7548635Z frame #9: <unknown function> + 0x94ac3 (0x7f13e5801ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7548849Z frame #10: <unknown function> + 0x126850 (0x7f13e5893850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7548853Z 
2025-07-17T11:07:26.7548945Z SIGABRT(6), PID: 1817871, Thread 1822051: 
2025-07-17T11:07:26.7549415Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7549691Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7549902Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7550129Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa3caac52dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7550608Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fa304f00741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7550842Z frame #5: <unknown function> + 0xdc253 (0x7fa2d742a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7551054Z frame #6: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7551264Z frame #7: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7551268Z 
2025-07-17T11:07:26.7551358Z SIGABRT(6), PID: 1817871, Thread 1822052: 
2025-07-17T11:07:26.7551831Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7552037Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7552203Z frame #2: ioctl + 0x3f (0x7fa3cab4b94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7552438Z frame #3: <unknown function> + 0x13eef0 (0x7fa246deeef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7552668Z frame #4: <unknown function> + 0x1378c4 (0x7fa246de78c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7552972Z frame #5: <unknown function> + 0x95b2b (0x7fa246d45b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7553200Z frame #6: <unknown function> + 0x34b71 (0x7fa246ce4b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7553414Z frame #7: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7553796Z frame #8: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7553800Z 
2025-07-17T11:07:26.7553892Z SIGABRT(6), PID: 1817871, Thread 1822143: 
2025-07-17T11:07:26.7554367Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7554655Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7554824Z frame #2: __poll + 0x4f (0x7fa3cab49bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7555045Z frame #3: <unknown function> + 0xd4540 (0x7fa28ef5e540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7555251Z frame #4: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7555460Z frame #5: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7555466Z 
2025-07-17T11:07:26.7555556Z SIGABRT(6), PID: 1817871, Thread 1822151: 
2025-07-17T11:07:26.7556031Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7556237Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7556406Z frame #2: __poll + 0x4f (0x7fa3cab49bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7556621Z frame #3: <unknown function> + 0xd65a2 (0x7fa28ef605a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7556834Z frame #4: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7557041Z frame #5: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7557045Z 
2025-07-17T11:07:26.7557134Z SIGABRT(6), PID: 1817871, Thread 1822163: 
2025-07-17T11:07:26.7557667Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7557881Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7558084Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7558290Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7558506Z frame #4: <unknown function> + 0xd1e7f (0x7fa28ef5be7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7558714Z frame #5: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7558920Z frame #6: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7558925Z 
2025-07-17T11:07:26.7559014Z SIGABRT(6), PID: 1817866, Thread 1821708: 
2025-07-17T11:07:26.7559487Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7559700Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7559880Z frame #2: epoll_wait + 0x5e (0x7fee20581e2e in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7560214Z frame #3: uv__io_poll + 0x6bd (0x7fed7113c44d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7560529Z frame #4: uv_run + 0x12a (0x7fed7113036a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7560987Z frame #5: <unknown function> + 0x658a08a (0x7fed654a408a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7561218Z frame #6: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7561429Z frame #7: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7561702Z frame #8: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7561706Z 
2025-07-17T11:07:26.7561799Z SIGABRT(6), PID: 1817871, Thread 1822170: 
2025-07-17T11:07:26.7562272Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7562540Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7562750Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7562961Z frame #3: <unknown function> + 0x9ca53 (0x7fa3caacda53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7563213Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x556e5a941bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7563442Z frame #5: <unknown function> + 0x1d314f (0x556e5aa0014f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7563669Z frame #6: <unknown function> + 0x1d2f6b (0x556e5a9fff6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7563890Z frame #7: <unknown function> + 0x6df36 (0x556e5a89af36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7564134Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7564363Z frame #9: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7564602Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7564831Z frame #11: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7565064Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7565289Z frame #13: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7565586Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7565817Z frame #15: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7566053Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7566277Z frame #17: <unknown function> + 0x147734 (0x556e5a974734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7566495Z frame #18: <unknown function> + 0x21ca82 (0x556e5aa49a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7566719Z frame #19: <unknown function> + 0x21c9e4 (0x556e5aa499e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7566933Z frame #20: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7567150Z frame #21: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7567242Z SIGABRT(6), PID: 1817869, Thread 1822198: 
2025-07-17T11:07:26.7567722Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7567931Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7568141Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7568341Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7568783Z frame #4: <unknown function> + 0x5a02023 (0x7ff9662c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7569349Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9662c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7570007Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9662c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7570392Z frame #7: <unknown function> + 0x79d655 (0x7ff9777a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7570678Z frame #8: <unknown function> + 0xdc253 (0x7ff92e7fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7570902Z frame #9: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7571116Z frame #10: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7571121Z 
2025-07-17T11:07:26.7571124Z 
2025-07-17T11:07:26.7571216Z SIGABRT(6), PID: 1817873, Thread 1822207: 
2025-07-17T11:07:26.7571691Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7571902Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7572103Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7572304Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7572674Z frame #4: <unknown function> + 0x5a02023 (0x7f48a6530023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7573232Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f48a65341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7573883Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f48a652b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7574273Z frame #7: <unknown function> + 0x79d655 (0x7f48b7a1b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7574502Z frame #8: <unknown function> + 0xdc253 (0x7f486ea66253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7574710Z frame #9: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7574921Z frame #10: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7574927Z 
2025-07-17T11:07:26.7575019Z SIGABRT(6), PID: 1817871, Thread 1822187: 
2025-07-17T11:07:26.7575493Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7575702Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7575869Z frame #2: read + 0x4c (0x7fa3cab4581c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7576311Z frame #3: _Py_read + 0x4c (0x556e5aa07a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7576543Z frame #4: <unknown function> + 0x208c3b (0x556e5aa35c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7576766Z frame #5: <unknown function> + 0x14648d (0x556e5a97348d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7577013Z frame #6: PyObject_VectorcallMethod + 0x85 (0x556e5a982215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7577341Z frame #7: <unknown function> + 0x208ae7 (0x556e5aa35ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7577558Z frame #8: <unknown function> + 0x208472 (0x556e5aa35472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7577779Z frame #9: <unknown function> + 0x1da6ee (0x556e5aa076ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7578079Z frame #10: <unknown function> + 0x1483bc (0x556e5a9753bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7578324Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7578547Z frame #12: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7578787Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x556e5a9593bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7579075Z frame #14: <unknown function> + 0x147734 (0x556e5a974734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7579334Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x556e5a95bc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7579558Z frame #16: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7579793Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7580017Z frame #18: _PyFunction_Vectorcall + 0x6c (0x556e5a9690ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7580257Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x556e5a9597b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7580481Z frame #20: <unknown function> + 0x147734 (0x556e5a974734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7580703Z frame #21: <unknown function> + 0x21ca82 (0x556e5aa49a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7580922Z frame #22: <unknown function> + 0x21c9e4 (0x556e5aa499e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7581139Z frame #23: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7581349Z frame #24: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7581353Z 
2025-07-17T11:07:26.7581447Z SIGABRT(6), PID: 1817866, Thread 1822127: 
2025-07-17T11:07:26.7582006Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7582228Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7582435Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7582663Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fee204f02dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7583109Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fed5a92e048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7583534Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fed5a930bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7583765Z frame #6: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7583975Z frame #7: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7584188Z frame #8: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7584195Z 
2025-07-17T11:07:26.7584285Z SIGABRT(6), PID: 1817866, Thread 1822128: 
2025-07-17T11:07:26.7584763Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7584975Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7585247Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7585470Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fee204f02dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7585948Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fed5a928741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.7586260Z frame #5: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7586470Z frame #6: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7586678Z frame #7: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7586682Z 
2025-07-17T11:07:26.7586774Z SIGABRT(6), PID: 1817866, Thread 1822130: 
2025-07-17T11:07:26.7587300Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7587512Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7587679Z frame #2: ioctl + 0x3f (0x7fee2057694f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7587912Z frame #3: <unknown function> + 0x13eef0 (0x7fec9c816ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7588147Z frame #4: <unknown function> + 0x1378c4 (0x7fec9c80f8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7588381Z frame #5: <unknown function> + 0x95b2b (0x7fec9c76db2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7588607Z frame #6: <unknown function> + 0x34b71 (0x7fec9c70cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7588817Z frame #7: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7589027Z frame #8: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7589033Z 
2025-07-17T11:07:26.7589122Z SIGABRT(6), PID: 1817866, Thread 1822142: 
2025-07-17T11:07:26.7589592Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7589802Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7590035Z frame #2: __poll + 0x4f (0x7fee20574bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7590255Z frame #3: <unknown function> + 0xd4540 (0x7fece4986540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7590462Z frame #4: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7590670Z frame #5: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7590676Z 
2025-07-17T11:07:26.7590766Z SIGABRT(6), PID: 1817866, Thread 1822150: 
2025-07-17T11:07:26.7591241Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7591448Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7591613Z frame #2: __poll + 0x4f (0x7fee20574bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7591830Z frame #3: <unknown function> + 0xd65a2 (0x7fece49885a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7592039Z frame #4: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7592246Z frame #5: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7592251Z 
2025-07-17T11:07:26.7592342Z SIGABRT(6), PID: 1817873, Thread 1822208: 
2025-07-17T11:07:26.7592807Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7593085Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7593288Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7593491Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7593930Z frame #4: <unknown function> + 0x5a02023 (0x7f48a6530023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7594499Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f48a65341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7595148Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f48a652b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7595536Z frame #7: <unknown function> + 0x79d655 (0x7f48b7a1b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7595764Z frame #8: <unknown function> + 0xdc253 (0x7f486ea66253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7595972Z frame #9: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7596187Z frame #10: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7596194Z 
2025-07-17T11:07:26.7596283Z SIGABRT(6), PID: 1817869, Thread 1822199: 
2025-07-17T11:07:26.7596755Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7596966Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7597172Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7597374Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7597747Z frame #4: <unknown function> + 0x5a02023 (0x7ff9662c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7598376Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9662c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7598973Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9662c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7599353Z frame #7: <unknown function> + 0x79d655 (0x7ff9777a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7599585Z frame #8: <unknown function> + 0xdc253 (0x7ff92e7fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7599796Z frame #9: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7600016Z frame #10: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7600020Z 
2025-07-17T11:07:26.7600109Z SIGABRT(6), PID: 1817866, Thread 1822159: 
2025-07-17T11:07:26.7600583Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7600793Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7601001Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7601202Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7601489Z frame #4: <unknown function> + 0xd1e7f (0x7fece4983e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.7601698Z frame #5: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7601910Z frame #6: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7601914Z 
2025-07-17T11:07:26.7602002Z SIGABRT(6), PID: 1817871, Thread 1822239: 
2025-07-17T11:07:26.7602535Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7602742Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7602953Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7603208Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7603590Z frame #4: <unknown function> + 0x5a02023 (0x7fa30eef4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7604151Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30eef81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7604748Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30eeef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7605128Z frame #7: <unknown function> + 0x79d655 (0x7fa3203df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7605357Z frame #8: <unknown function> + 0xdc253 (0x7fa2d742a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7605567Z frame #9: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7605788Z frame #10: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7605792Z 
2025-07-17T11:07:26.7605881Z SIGABRT(6), PID: 1817866, Thread 1822164: 
2025-07-17T11:07:26.7606357Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7606633Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7606839Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7607047Z frame #3: <unknown function> + 0x9ca53 (0x7fee204f8a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7607302Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x562254996bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7607530Z frame #5: <unknown function> + 0x1d314f (0x562254a5514f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7607753Z frame #6: <unknown function> + 0x1d2f6b (0x562254a54f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7607972Z frame #7: <unknown function> + 0x6df36 (0x5622548eff36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7608214Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7608442Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7608682Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7608905Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7609139Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7609367Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7609667Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7609890Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7610124Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7610339Z frame #17: <unknown function> + 0x147734 (0x5622549c9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7610629Z frame #18: <unknown function> + 0x21ca82 (0x562254a9ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7610847Z frame #19: <unknown function> + 0x21c9e4 (0x562254a9e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7611065Z frame #20: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7611335Z frame #21: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7611345Z 
2025-07-17T11:07:26.7611437Z SIGABRT(6), PID: 1817866, Thread 1822173: 
2025-07-17T11:07:26.7611911Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7612120Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7612292Z frame #2: read + 0x4c (0x7fee2057081c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7612473Z frame #3: _Py_read + 0x4c (0x562254a5ca6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7612698Z frame #4: <unknown function> + 0x208c3b (0x562254a8ac3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7612919Z frame #5: <unknown function> + 0x14648d (0x5622549c848d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7613166Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5622549d7215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7613386Z frame #7: <unknown function> + 0x208ae7 (0x562254a8aae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7613602Z frame #8: <unknown function> + 0x208472 (0x562254a8a472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7613822Z frame #9: <unknown function> + 0x1da6ee (0x562254a5c6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7614042Z frame #10: <unknown function> + 0x1483bc (0x5622549ca3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7614346Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7614574Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7614812Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5622549ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7615033Z frame #14: <unknown function> + 0x147734 (0x5622549c9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7615278Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5622549b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7615503Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7615735Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7616058Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5622549be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7616295Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5622549ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7616511Z frame #20: <unknown function> + 0x147734 (0x5622549c9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7616726Z frame #21: <unknown function> + 0x21ca82 (0x562254a9ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7616947Z frame #22: <unknown function> + 0x21c9e4 (0x562254a9e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7617161Z frame #23: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7617484Z frame #24: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7617489Z 
2025-07-17T11:07:26.7617579Z SIGABRT(6), PID: 1817873, Thread 1822209: 
2025-07-17T11:07:26.7618055Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7618345Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7618551Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7618750Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7619201Z frame #4: <unknown function> + 0x5a02023 (0x7f48a6530023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7619769Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f48a65341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7620358Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f48a652b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7620741Z frame #7: <unknown function> + 0x79d655 (0x7f48b7a1b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7620980Z frame #8: <unknown function> + 0xdc253 (0x7f486ea66253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7621185Z frame #9: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7621403Z frame #10: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7621409Z 
2025-07-17T11:07:26.7621499Z SIGABRT(6), PID: 1817869, Thread 1822200: 
2025-07-17T11:07:26.7621971Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7622180Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7622386Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7622687Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7623067Z frame #4: <unknown function> + 0x5a02023 (0x7ff9662c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7623630Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9662c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7624226Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9662c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7624604Z frame #7: <unknown function> + 0x79d655 (0x7ff9777a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7624841Z frame #8: <unknown function> + 0xdc253 (0x7ff92e7fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7625052Z frame #9: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7625264Z frame #10: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7625268Z 
2025-07-17T11:07:26.7625361Z SIGABRT(6), PID: 1817871, Thread 1822241: 
2025-07-17T11:07:26.7625839Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7626123Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7626329Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7626533Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7626969Z frame #4: <unknown function> + 0x5a02023 (0x7fa30eef4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7627534Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30eef81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7628181Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30eeef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7628568Z frame #7: <unknown function> + 0x79d655 (0x7fa3203df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7628797Z frame #8: <unknown function> + 0xdc253 (0x7fa2d742a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7629009Z frame #9: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7629228Z frame #10: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7629232Z 
2025-07-17T11:07:26.7629327Z SIGABRT(6), PID: 1817866, Thread 1822188: 
2025-07-17T11:07:26.7629799Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7630016Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7630223Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7630425Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7630793Z frame #4: <unknown function> + 0x5a02023 (0x7fed6491c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7631417Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed649201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7632009Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed649171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7632396Z frame #7: <unknown function> + 0x79d655 (0x7fed75e07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7632626Z frame #8: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7632835Z frame #9: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7633047Z frame #10: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7633051Z 
2025-07-17T11:07:26.7633147Z SIGABRT(6), PID: 1817873, Thread 1822210: 
2025-07-17T11:07:26.7633616Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7633830Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7634032Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7634232Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7634672Z frame #4: <unknown function> + 0x5a02023 (0x7f48a6530023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7635235Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f48a65341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7635900Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f48a652b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7636284Z frame #7: <unknown function> + 0x79d655 (0x7f48b7a1b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7636579Z frame #8: <unknown function> + 0xdc253 (0x7f486ea66253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7636795Z frame #9: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7637013Z frame #10: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7637017Z 
2025-07-17T11:07:26.7637107Z SIGABRT(6), PID: 1817869, Thread 1822201: 
2025-07-17T11:07:26.7637592Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7637805Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7638012Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7638211Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7638583Z frame #4: <unknown function> + 0x5a02023 (0x7ff9662c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7639141Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9662c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7639731Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9662c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7640173Z frame #7: <unknown function> + 0x79d655 (0x7ff9777a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7640411Z frame #8: <unknown function> + 0xdc253 (0x7ff92e7fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7640621Z frame #9: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7640840Z frame #10: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7640844Z 
2025-07-17T11:07:26.7640937Z SIGABRT(6), PID: 1817871, Thread 1822243: 
2025-07-17T11:07:26.7641422Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7641628Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7641836Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7642039Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7642413Z frame #4: <unknown function> + 0x5a02023 (0x7fa30eef4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7642973Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30eef81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7643631Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30eeef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7644007Z frame #7: <unknown function> + 0x79d655 (0x7fa3203df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7644304Z frame #8: <unknown function> + 0xdc253 (0x7fa2d742a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7644513Z frame #9: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7644731Z frame #10: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7644735Z 
2025-07-17T11:07:26.7644830Z SIGABRT(6), PID: 1817866, Thread 1822189: 
2025-07-17T11:07:26.7645363Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7645579Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7645783Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7645988Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7646358Z frame #4: <unknown function> + 0x5a02023 (0x7fed6491c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7646915Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed649201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7647502Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed649171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7647882Z frame #7: <unknown function> + 0x79d655 (0x7fed75e07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7648114Z frame #8: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7648326Z frame #9: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7648624Z frame #10: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7648628Z 
2025-07-17T11:07:26.7648724Z SIGABRT(6), PID: 1817873, Thread 1822211: 
2025-07-17T11:07:26.7649192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f48707f9b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7649401Z frame #1: <unknown function> + 0x42520 (0x7f49620b5520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7649603Z frame #2: <unknown function> + 0x91117 (0x7f4962104117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7649801Z frame #3: pthread_cond_wait + 0x211 (0x7f4962106a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7650171Z frame #4: <unknown function> + 0x5a02023 (0x7f48a6530023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7650732Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f48a65341ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7651331Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f48a652b1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7651714Z frame #7: <unknown function> + 0x79d655 (0x7f48b7a1b655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7652010Z frame #8: <unknown function> + 0xdc253 (0x7f486ea66253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7652220Z frame #9: <unknown function> + 0x94ac3 (0x7f4962107ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7652431Z frame #10: <unknown function> + 0x126850 (0x7f4962199850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7652523Z SIGABRT(6), PID: 1817869, Thread 1822202: 
2025-07-17T11:07:26.7653060Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7653275Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7653480Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7653741Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7654115Z frame #4: <unknown function> + 0x5a02023 (0x7ff9662c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7654670Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9662c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7655263Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9662c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7655638Z frame #7: <unknown function> + 0x79d655 (0x7ff9777a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7655870Z frame #8: <unknown function> + 0xdc253 (0x7ff92e7fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7656147Z frame #9: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7656367Z frame #10: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7656371Z 
2025-07-17T11:07:26.7656374Z 
2025-07-17T11:07:26.7656464Z SIGABRT(6), PID: 1817871, Thread 1822245: 
2025-07-17T11:07:26.7656940Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7657232Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7657447Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7657652Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7658034Z frame #4: <unknown function> + 0x5a02023 (0x7fa30eef4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7658595Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30eef81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7659190Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30eeef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7659573Z frame #7: <unknown function> + 0x79d655 (0x7fa3203df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7659805Z frame #8: <unknown function> + 0xdc253 (0x7fa2d742a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7660011Z frame #9: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7660230Z frame #10: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7660234Z 
2025-07-17T11:07:26.7660402Z SIGABRT(6), PID: 1817866, Thread 1822190: 
2025-07-17T11:07:26.7660877Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7661087Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7661294Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7661572Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7661945Z frame #4: <unknown function> + 0x5a02023 (0x7fed6491c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7662577Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed649201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7663173Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed649171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7663551Z frame #7: <unknown function> + 0x79d655 (0x7fed75e07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7663780Z frame #8: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7663991Z frame #9: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7664207Z frame #10: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7664211Z 
2025-07-17T11:07:26.7664303Z SIGABRT(6), PID: 1817869, Thread 1822203: 
2025-07-17T11:07:26.7664778Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff93058eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7664988Z frame #1: <unknown function> + 0x42520 (0x7ffa21e45520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7665193Z frame #2: <unknown function> + 0x91117 (0x7ffa21e94117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7665394Z frame #3: pthread_cond_wait + 0x211 (0x7ffa21e96a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7665827Z frame #4: <unknown function> + 0x5a02023 (0x7ff9662c5023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7666391Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff9662c91ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7666979Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff9662c01d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7667358Z frame #7: <unknown function> + 0x79d655 (0x7ff9777a7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7667588Z frame #8: <unknown function> + 0xdc253 (0x7ff92e7fb253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7667799Z frame #9: <unknown function> + 0x94ac3 (0x7ffa21e97ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7668015Z frame #10: <unknown function> + 0x126850 (0x7ffa21f29850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7668020Z 
2025-07-17T11:07:26.7668114Z SIGABRT(6), PID: 1817871, Thread 1822247: 
2025-07-17T11:07:26.7668584Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7668796Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7669002Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7669270Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7669639Z frame #4: <unknown function> + 0x5a02023 (0x7fa30eef4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7670201Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30eef81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7670849Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30eeef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7671284Z frame #7: <unknown function> + 0x79d655 (0x7fa3203df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7671520Z frame #8: <unknown function> + 0xdc253 (0x7fa2d742a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7671737Z frame #9: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7671955Z frame #10: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7671962Z 
2025-07-17T11:07:26.7672053Z SIGABRT(6), PID: 1817866, Thread 1822191: 
2025-07-17T11:07:26.7672535Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7672742Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7672951Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7673149Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7673520Z frame #4: <unknown function> + 0x5a02023 (0x7fed6491c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7674072Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed649201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7674725Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed649171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7675106Z frame #7: <unknown function> + 0x79d655 (0x7fed75e07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7675342Z frame #8: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7675552Z frame #9: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7675769Z frame #10: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7675774Z 
2025-07-17T11:07:26.7675863Z SIGABRT(6), PID: 1817871, Thread 1822249: 
2025-07-17T11:07:26.7676335Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7676546Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7676759Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7676959Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7677329Z frame #4: <unknown function> + 0x5a02023 (0x7fa30eef4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7677889Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30eef81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7678549Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30eeef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7678990Z frame #7: <unknown function> + 0x79d655 (0x7fa3203df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7679221Z frame #8: <unknown function> + 0xdc253 (0x7fa2d742a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7679433Z frame #9: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7679727Z frame #10: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7679732Z 
2025-07-17T11:07:26.7679825Z SIGABRT(6), PID: 1817866, Thread 1822192: 
2025-07-17T11:07:26.7680304Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7680513Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7680720Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7680925Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7681297Z frame #4: <unknown function> + 0x5a02023 (0x7fed6491c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7681859Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed649201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7682446Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed649171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7682829Z frame #7: <unknown function> + 0x79d655 (0x7fed75e07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7683061Z frame #8: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7683336Z frame #9: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7683555Z frame #10: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7683559Z 
2025-07-17T11:07:26.7683658Z SIGABRT(6), PID: 1817871, Thread 1822250: 
2025-07-17T11:07:26.7684125Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7684336Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7684541Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7684744Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7685112Z frame #4: <unknown function> + 0x5a02023 (0x7fa30eef4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7685668Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30eef81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7686258Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30eeef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7686821Z frame #7: <unknown function> + 0x79d655 (0x7fa3203df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7687049Z frame #8: <unknown function> + 0xdc253 (0x7fa2d742a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7687263Z frame #9: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7687542Z frame #10: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7687547Z 
2025-07-17T11:07:26.7687639Z SIGABRT(6), PID: 1817866, Thread 1822193: 
2025-07-17T11:07:26.7688111Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7688378Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7688586Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7688794Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7689163Z frame #4: <unknown function> + 0x5a02023 (0x7fed6491c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7689721Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed649201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7690309Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed649171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7690687Z frame #7: <unknown function> + 0x79d655 (0x7fed75e07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7690919Z frame #8: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7691130Z frame #9: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7691348Z frame #10: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7691352Z 
2025-07-17T11:07:26.7691442Z SIGABRT(6), PID: 1817871, Thread 1822251: 
2025-07-17T11:07:26.7691984Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d91bdb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7692195Z frame #1: <unknown function> + 0x42520 (0x7fa3caa73520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7692405Z frame #2: <unknown function> + 0x91117 (0x7fa3caac2117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7692608Z frame #3: pthread_cond_wait + 0x211 (0x7fa3caac4a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7692979Z frame #4: <unknown function> + 0x5a02023 (0x7fa30eef4023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7693536Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30eef81ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7694126Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30eeef1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7694502Z frame #7: <unknown function> + 0x79d655 (0x7fa3203df655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7694732Z frame #8: <unknown function> + 0xdc253 (0x7fa2d742a253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7694941Z frame #9: <unknown function> + 0x94ac3 (0x7fa3caac5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7695228Z frame #10: <unknown function> + 0x126850 (0x7fa3cab57850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7695231Z 
2025-07-17T11:07:26.7695323Z SIGABRT(6), PID: 1817866, Thread 1822194: 
2025-07-17T11:07:26.7695796Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7696136Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7696347Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7696550Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7696999Z frame #4: <unknown function> + 0x5a02023 (0x7fed6491c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7697564Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed649201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7698158Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed649171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7698538Z frame #7: <unknown function> + 0x79d655 (0x7fed75e07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7698773Z frame #8: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7698982Z frame #9: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7699199Z frame #10: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7699204Z 
2025-07-17T11:07:26.7699294Z SIGABRT(6), PID: 1817866, Thread 1822195: 
2025-07-17T11:07:26.7699767Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fed2ebe5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7699983Z frame #1: <unknown function> + 0x42520 (0x7fee2049e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7700186Z frame #2: <unknown function> + 0x91117 (0x7fee204ed117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7700466Z frame #3: pthread_cond_wait + 0x211 (0x7fee204efa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7700834Z frame #4: <unknown function> + 0x5a02023 (0x7fed6491c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7701395Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed649201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7701983Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed649171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.7702363Z frame #7: <unknown function> + 0x79d655 (0x7fed75e07655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7702592Z frame #8: <unknown function> + 0xdc253 (0x7fed2ce52253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.7702807Z frame #9: <unknown function> + 0x94ac3 (0x7fee204f0ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7703019Z frame #10: <unknown function> + 0x126850 (0x7fee20582850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7703023Z 
2025-07-17T11:07:26.7703126Z ('RERUN', {'yellow': True}) [25.6502s] [100%]
2025-07-17T11:07:26.7703777Z distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph I0717 11:03:00.408000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1822836
2025-07-17T11:07:26.7704218Z I0717 11:03:00.409000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1822837
2025-07-17T11:07:26.7704566Z I0717 11:03:00.410000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1822838
2025-07-17T11:07:26.7704988Z I0717 11:03:00.410000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1822839
2025-07-17T11:07:26.7705331Z I0717 11:03:00.411000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1822840
2025-07-17T11:07:26.7705673Z I0717 11:03:00.412000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1822841
2025-07-17T11:07:26.7706093Z I0717 11:03:00.413000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1822842
2025-07-17T11:07:26.7706442Z I0717 11:03:00.414000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1822843
2025-07-17T11:07:26.7707449Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.7707540Z   import pkg_resources
2025-07-17T11:07:26.7708515Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.7708598Z   import pkg_resources
2025-07-17T11:07:26.7709568Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.7709647Z   import pkg_resources
2025-07-17T11:07:26.7710682Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.7710765Z   import pkg_resources
2025-07-17T11:07:26.7711736Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.7711815Z   import pkg_resources
2025-07-17T11:07:26.7712792Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.7712872Z   import pkg_resources
2025-07-17T11:07:26.7713843Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.7713921Z   import pkg_resources
2025-07-17T11:07:26.7714895Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.7715037Z   import pkg_resources
2025-07-17T11:07:26.7715526Z [rank1]:W0717 11:03:22.221000 1822837 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.7716081Z [rank4]:W0717 11:03:22.222000 1822840 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.7716610Z [rank5]:W0717 11:03:22.223000 1822841 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.7717079Z [rank6]:W0717 11:03:22.223000 1822842 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.7717545Z [rank0]:W0717 11:03:22.223000 1822836 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.7718014Z [rank7]:W0717 11:03:22.225000 1822843 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.7718477Z [rank2]:W0717 11:03:22.225000 1822838 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.7718944Z [rank3]:W0717 11:03:22.225000 1822839 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.7719229Z Memory access fault by GPU node-4 (Agent handle: 0x55a5661d0610) on address 0x7fed61a1d000. Reason: Unknown.
2025-07-17T11:07:26.7719507Z Memory access fault by GPU node-3 (Agent handle: 0x560b2610d240) on address 0x7f02cf41d000. Reason: Unknown.
2025-07-17T11:07:26.7719773Z Memory access fault by GPU node-6 (Agent handle: 0x5557c11f90d0) on address 0x7f9a0aa1f000. Reason: Unknown.
2025-07-17T11:07:26.7720041Z Memory access fault by GPU node-9 (Agent handle: 0x56103c4c1c40) on address 0x7f1349e1e000. Reason: Unknown.
2025-07-17T11:07:26.7720373Z Memory access fault by GPU node-7 (Agent handle: 0x555c1ab917d0) on address 0x7f76ebe04000. Reason: Unknown.
2025-07-17T11:07:26.7720648Z Memory access fault by GPU node-2 (Agent handle: 0x5627794b1410) on address 0x7fd72f41e000. Reason: Unknown.
2025-07-17T11:07:26.7720912Z Memory access fault by GPU node-5 (Agent handle: 0x55b31a3e1120) on address 0x7f634341d000. Reason: Unknown.
2025-07-17T11:07:26.7721182Z Memory access fault by GPU node-8 (Agent handle: 0x562d2d28d920) on address 0x7ef8af41d000. Reason: Unknown.
2025-07-17T11:07:26.7721280Z SIGABRT(6), PID: 1822837, Thread 1822837: 
2025-07-17T11:07:26.7721766Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7721985Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7722230Z frame #2: <unknown function> + 0x7743a (0x7f0b0b34b43a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7722461Z frame #3: <unknown function> + 0x76c8e (0x7f0b0b34ac8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7722689Z frame #4: <unknown function> + 0x6ad69 (0x7f0b0b33ed69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7722905Z frame #5: <unknown function> + 0x3f54db (0x7f0b9bfa34db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7723123Z frame #6: <unknown function> + 0x3e11d8 (0x7f0b9bf8f1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7728556Z frame #7: <unknown function> + 0xb2f86 (0x7f0b9bc60f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7728849Z frame #8: <unknown function> + 0x8e501 (0x7f0b9bc3c501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7729270Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f0bc6a437ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.7729808Z frame #10: <unknown function> + 0xd307d9 (0x7f0be4f967d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7730051Z frame #11: <unknown function> + 0x139902 (0x560b1be17902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7730310Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x560b1be0ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7730613Z frame #13: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7730862Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x560b1be0ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7731088Z frame #15: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7731332Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7731553Z frame #17: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7731797Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7732016Z frame #19: <unknown function> + 0x1474b2 (0x560b1be254b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7732232Z frame #20: PyObject_Call + 0xbc (0x560b1be25e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7732461Z frame #21: <unknown function> + 0x1ff69d (0x560b1bedd69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7732687Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x560b1be13023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7732932Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x560b1be0ecc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7733158Z frame #24: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7733400Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x560b1be0a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7733686Z frame #26: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7733921Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x560b1be0a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7734142Z frame #28: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7734382Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x560b1be0ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7734634Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x560b1be124b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7734854Z frame #31: _PyObject_Call_Prepend + 0x69 (0x560b1be23ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7735073Z frame #32: <unknown function> + 0x205949 (0x560b1bee3949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7735294Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x560b1be13023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7735538Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x560b1be0ecc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7735759Z frame #35: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7736072Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x560b1be0a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7736295Z frame #37: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7736532Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x560b1be0a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7736840Z frame #39: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7737075Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x560b1be0a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7737293Z frame #41: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7737613Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x560b1be0b3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7737835Z frame #43: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7738231Z frame #44: <unknown function> + 0x7ab3f7 (0x7f0be4a113f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7738521Z frame #45: <unknown function> + 0x13bc88 (0x560b1be19c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7738736Z frame #46: PyObject_Call + 0x20f (0x560b1be25f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7738976Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x560b1be0f98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7739195Z frame #48: <unknown function> + 0x147608 (0x560b1be25608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7739434Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x560b1be0cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7739662Z frame #50: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7739896Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x560b1be0a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7740117Z frame #52: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7740359Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x560b1be0b3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7740582Z frame #54: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7740816Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x560b1be0a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7741037Z frame #56: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7741284Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x560b1be0cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7741585Z frame #58: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7741823Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x560b1be0cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7742039Z frame #60: <unknown function> + 0x147608 (0x560b1be25608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7742273Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x560b1be0cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7742493Z frame #62: <unknown function> + 0x147608 (0x560b1be25608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7742501Z 
2025-07-17T11:07:26.7742599Z SIGABRT(6), PID: 1822837, Thread 1822876: 
2025-07-17T11:07:26.7743102Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7743319Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7743530Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7743739Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7744279Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7744496Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7744785Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7744790Z 
2025-07-17T11:07:26.7744888Z SIGABRT(6), PID: 1822837, Thread 1822879: 
2025-07-17T11:07:26.7745376Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7745651Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7745861Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7746067Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7746642Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7746854Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7747071Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7747076Z 
2025-07-17T11:07:26.7747167Z SIGABRT(6), PID: 1822837, Thread 1822883: 
2025-07-17T11:07:26.7747639Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7747846Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7748050Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7748248Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7748752Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7748962Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7749168Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7749172Z 
2025-07-17T11:07:26.7749265Z SIGABRT(6), PID: 1822837, Thread 1822885: 
2025-07-17T11:07:26.7749799Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7750012Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7750215Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7750413Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7750908Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7751119Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7751324Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7751328Z 
2025-07-17T11:07:26.7751419Z SIGABRT(6), PID: 1822837, Thread 1822887: 
2025-07-17T11:07:26.7751888Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7752094Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7752295Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7752494Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7753097Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7753305Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7753510Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7753578Z 
2025-07-17T11:07:26.7753671Z SIGABRT(6), PID: 1822837, Thread 1822889: 
2025-07-17T11:07:26.7754136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7754343Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7754606Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7754808Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7755307Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7755514Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7755721Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7755726Z 
2025-07-17T11:07:26.7755819Z SIGABRT(6), PID: 1822837, Thread 1822890: 
2025-07-17T11:07:26.7756283Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7756490Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7756695Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7756896Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7757386Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7757591Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7757863Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7757868Z 
2025-07-17T11:07:26.7757967Z SIGABRT(6), PID: 1822837, Thread 1822893: 
2025-07-17T11:07:26.7758434Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7758641Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7758843Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7759043Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7759535Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7759739Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7759948Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7759952Z 
2025-07-17T11:07:26.7760039Z SIGABRT(6), PID: 1822837, Thread 1822894: 
2025-07-17T11:07:26.7760521Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7760729Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7761009Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7761208Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7761712Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7761985Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7762197Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7762201Z 
2025-07-17T11:07:26.7762290Z SIGABRT(6), PID: 1822837, Thread 1822897: 
2025-07-17T11:07:26.7762823Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7763033Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7763236Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7763433Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7763928Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7764136Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7764346Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7764351Z 
2025-07-17T11:07:26.7764439Z SIGABRT(6), PID: 1822837, Thread 1822899: 
2025-07-17T11:07:26.7764906Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7765111Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7765316Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7765510Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7766073Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7766283Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7766494Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7766498Z 
2025-07-17T11:07:26.7766587Z SIGABRT(6), PID: 1822837, Thread 1822900: 
2025-07-17T11:07:26.7767054Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7767260Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7767467Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7767662Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7768163Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7768369Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7768578Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7768582Z 
2025-07-17T11:07:26.7768670Z SIGABRT(6), PID: 1822837, Thread 1822902: 
2025-07-17T11:07:26.7769208Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7769413Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7769614Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7769879Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7770368Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7770577Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7770844Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7770851Z 
2025-07-17T11:07:26.7770944Z SIGABRT(6), PID: 1822837, Thread 1822904: 
2025-07-17T11:07:26.7771412Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7771617Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7771821Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7772019Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7772509Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7772718Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7772928Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7772935Z 
2025-07-17T11:07:26.7773026Z SIGABRT(6), PID: 1822837, Thread 1822906: 
2025-07-17T11:07:26.7773491Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7773696Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7773965Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7774168Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7774662Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7774870Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7775080Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7775084Z 
2025-07-17T11:07:26.7775174Z SIGABRT(6), PID: 1822837, Thread 1822908: 
2025-07-17T11:07:26.7775637Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7775846Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7776115Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7776316Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7776807Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7777102Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7777312Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7777316Z 
2025-07-17T11:07:26.7777410Z SIGABRT(6), PID: 1822837, Thread 1822910: 
2025-07-17T11:07:26.7777882Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7778164Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7778367Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7778566Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7779130Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7779344Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7779552Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7779559Z 
2025-07-17T11:07:26.7779648Z SIGABRT(6), PID: 1822837, Thread 1822912: 
2025-07-17T11:07:26.7780121Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7780324Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7780527Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7780723Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7781220Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7781426Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7781637Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7781641Z 
2025-07-17T11:07:26.7781730Z SIGABRT(6), PID: 1822837, Thread 1822914: 
2025-07-17T11:07:26.7782299Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7782507Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7782710Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7782911Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7783417Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7783623Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7783833Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7783839Z 
2025-07-17T11:07:26.7783929Z SIGABRT(6), PID: 1822837, Thread 1822917: 
2025-07-17T11:07:26.7784396Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7784598Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7784803Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7785068Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7785560Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7785768Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7786043Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7786047Z 
2025-07-17T11:07:26.7786135Z SIGABRT(6), PID: 1822837, Thread 1822919: 
2025-07-17T11:07:26.7786606Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7786870Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7787080Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7787276Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7787772Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7787979Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7788192Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7788196Z 
2025-07-17T11:07:26.7788285Z SIGABRT(6), PID: 1822840, Thread 1822840: 
2025-07-17T11:07:26.7788757Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7788962Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7789201Z frame #2: <unknown function> + 0x7743a (0x7fa24705d43a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7789427Z frame #3: <unknown function> + 0x76c8e (0x7fa24705cc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7789655Z frame #4: <unknown function> + 0x6ad69 (0x7fa247050d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7789870Z frame #5: <unknown function> + 0x3f54db (0x7fa2d7cb54db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7790153Z frame #6: <unknown function> + 0x3e11d8 (0x7fa2d7ca11d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7790371Z frame #7: <unknown function> + 0xb2f86 (0x7fa2d7972f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7790579Z frame #8: <unknown function> + 0x8e501 (0x7fa2d794e501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7790970Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fa3027557ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.7791361Z frame #10: <unknown function> + 0xd307d9 (0x7fa320ca87d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7791590Z frame #11: <unknown function> + 0x139902 (0x5557b748b902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7791843Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x5557b7482a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7792078Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7792321Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x5557b7482a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7792545Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7792786Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7793006Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7793312Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7793532Z frame #19: <unknown function> + 0x1474b2 (0x5557b74994b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7793739Z frame #20: PyObject_Call + 0xbc (0x5557b7499e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7794110Z frame #21: <unknown function> + 0x1ff69d (0x5557b755169d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7794338Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x5557b7487023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7794581Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x5557b7482cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7794806Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7795103Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5557b747e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7795333Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7795566Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5557b747e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7795786Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7796029Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x5557b7482a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7796284Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x5557b74864b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7796498Z frame #31: _PyObject_Call_Prepend + 0x69 (0x5557b7497ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7796716Z frame #32: <unknown function> + 0x205949 (0x5557b7557949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7796938Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x5557b7487023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7797179Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x5557b7482cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7797400Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7797632Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5557b747e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7797853Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7798149Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5557b747e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7798371Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7798606Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5557b747e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7798828Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7799070Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5557b747f3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7799291Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7799675Z frame #44: <unknown function> + 0x7ab3f7 (0x7fa3207233f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7799898Z frame #45: <unknown function> + 0x13bc88 (0x5557b748dc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7800105Z frame #46: PyObject_Call + 0x20f (0x5557b7499f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7800345Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x5557b748398d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7800565Z frame #48: <unknown function> + 0x147608 (0x5557b7499608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7800804Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x5557b7480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7801097Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7801333Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5557b747e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7801554Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7801855Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5557b747f3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7802077Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7802309Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5557b747e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7802529Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7802826Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x5557b7480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7803054Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7803286Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x5557b7480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7803502Z frame #60: <unknown function> + 0x147608 (0x5557b7499608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7803735Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x5557b7480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7803956Z frame #62: <unknown function> + 0x147608 (0x5557b7499608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7803960Z 
2025-07-17T11:07:26.7804051Z SIGABRT(6), PID: 1822837, Thread 1822921: 
2025-07-17T11:07:26.7804532Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7804746Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7804956Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7805159Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7805663Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7805938Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7806158Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7806163Z 
2025-07-17T11:07:26.7806256Z SIGABRT(6), PID: 1822840, Thread 1822990: 
2025-07-17T11:07:26.7806729Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7806939Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7807142Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7807342Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7807839Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7808051Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7808259Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7808263Z 
2025-07-17T11:07:26.7808354Z SIGABRT(6), PID: 1822837, Thread 1822922: 
2025-07-17T11:07:26.7808817Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7809094Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7809298Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7809496Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7810069Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7810285Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7810496Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7810500Z 
2025-07-17T11:07:26.7810671Z SIGABRT(6), PID: 1822840, Thread 1822994: 
2025-07-17T11:07:26.7811145Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7811357Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7811561Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7811763Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7812263Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7812474Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7812685Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7812690Z 
2025-07-17T11:07:26.7812782Z SIGABRT(6), PID: 1822837, Thread 1822924: 
2025-07-17T11:07:26.7813247Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7813454Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7813654Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7813919Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7814417Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7814628Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7814834Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7814841Z 
2025-07-17T11:07:26.7814932Z SIGABRT(6), PID: 1822840, Thread 1822999: 
2025-07-17T11:07:26.7815394Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7815599Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7815805Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7816071Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7816564Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7816776Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7817077Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7817081Z 
2025-07-17T11:07:26.7817170Z SIGABRT(6), PID: 1822837, Thread 1822926: 
2025-07-17T11:07:26.7817638Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7817845Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7818131Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7818329Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7818900Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7819113Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7819327Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7819331Z 
2025-07-17T11:07:26.7819419Z SIGABRT(6), PID: 1822840, Thread 1823003: 
2025-07-17T11:07:26.7819885Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7820093Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7820298Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7820495Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7820994Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7821201Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7821410Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7821414Z 
2025-07-17T11:07:26.7821502Z SIGABRT(6), PID: 1822837, Thread 1822928: 
2025-07-17T11:07:26.7822160Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7822369Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7822575Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7822771Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7823263Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7823472Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7823681Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7823685Z 
2025-07-17T11:07:26.7823773Z SIGABRT(6), PID: 1822840, Thread 1823007: 
2025-07-17T11:07:26.7824249Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7824455Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7824660Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7824859Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7825348Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7825624Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7825835Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7825897Z 
2025-07-17T11:07:26.7825989Z SIGABRT(6), PID: 1822837, Thread 1822930: 
2025-07-17T11:07:26.7826456Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7826660Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7826921Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7827120Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7827621Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7827829Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7828036Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7828126Z SIGABRT(6), PID: 1822840, Thread 1823011: 
2025-07-17T11:07:26.7828588Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7828794Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7828999Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7829198Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7829692Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7829905Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7830116Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7830184Z 
2025-07-17T11:07:26.7830189Z 
2025-07-17T11:07:26.7830282Z SIGABRT(6), PID: 1822840, Thread 1823015: 
2025-07-17T11:07:26.7830743Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7830954Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7831160Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7831361Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7831857Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7832067Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7832277Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7832281Z 
2025-07-17T11:07:26.7832372Z SIGABRT(6), PID: 1822837, Thread 1822932: 
2025-07-17T11:07:26.7832834Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7833041Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7833314Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7833515Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7834002Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7834276Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7834482Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7834573Z SIGABRT(6), PID: 1822840, Thread 1823019: 
2025-07-17T11:07:26.7835094Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7835304Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7835509Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7835710Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7836198Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7836412Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7836622Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7836626Z 
2025-07-17T11:07:26.7836629Z 
2025-07-17T11:07:26.7836721Z SIGABRT(6), PID: 1822840, Thread 1823024: 
2025-07-17T11:07:26.7837186Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7837397Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7837603Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7837801Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7838360Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7838572Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7838784Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7838788Z 
2025-07-17T11:07:26.7838879Z SIGABRT(6), PID: 1822837, Thread 1822934: 
2025-07-17T11:07:26.7839345Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7839551Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7839754Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7839948Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7840450Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7840656Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7840867Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7840871Z 
2025-07-17T11:07:26.7840960Z SIGABRT(6), PID: 1822840, Thread 1823028: 
2025-07-17T11:07:26.7841502Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7841708Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7841913Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7842202Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7842702Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7842909Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7843182Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7843190Z 
2025-07-17T11:07:26.7843281Z SIGABRT(6), PID: 1822837, Thread 1822937: 
2025-07-17T11:07:26.7843751Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7843954Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7844160Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7844360Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7844855Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7845064Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7845274Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7845280Z 
2025-07-17T11:07:26.7845366Z SIGABRT(6), PID: 1822840, Thread 1823031: 
2025-07-17T11:07:26.7845836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7846043Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7846318Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7846522Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7847020Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7847228Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7847440Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7847445Z 
2025-07-17T11:07:26.7847532Z SIGABRT(6), PID: 1822837, Thread 1822940: 
2025-07-17T11:07:26.7847999Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7848206Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7848407Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7848603Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7849094Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7849371Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7849579Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7849583Z 
2025-07-17T11:07:26.7849672Z SIGABRT(6), PID: 1822840, Thread 1823035: 
2025-07-17T11:07:26.7850136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7850409Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7850613Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7850812Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7851361Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7851578Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7851785Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7851789Z 
2025-07-17T11:07:26.7851879Z SIGABRT(6), PID: 1822837, Thread 1822942: 
2025-07-17T11:07:26.7852347Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7852554Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7852755Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7852953Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7853447Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7853658Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7853865Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7853869Z 
2025-07-17T11:07:26.7853960Z SIGABRT(6), PID: 1822840, Thread 1823039: 
2025-07-17T11:07:26.7854482Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7854693Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7854895Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7855099Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7855596Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7855804Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7856087Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7856182Z SIGABRT(6), PID: 1822837, Thread 1822945: 
2025-07-17T11:07:26.7856647Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7856853Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7857057Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7857256Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7857835Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7858045Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7858252Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7858329Z 
2025-07-17T11:07:26.7858338Z 
2025-07-17T11:07:26.7858429Z SIGABRT(6), PID: 1822837, Thread 1822948: 
2025-07-17T11:07:26.7858897Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7859172Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7859379Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7859579Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7860072Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7860280Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7860491Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7860495Z 
2025-07-17T11:07:26.7860582Z SIGABRT(6), PID: 1822840, Thread 1823043: 
2025-07-17T11:07:26.7861046Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7861255Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7861463Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7861659Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7862156Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7862441Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7862655Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7862659Z 
2025-07-17T11:07:26.7862745Z SIGABRT(6), PID: 1822837, Thread 1822951: 
2025-07-17T11:07:26.7863215Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7863419Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7863629Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7863826Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7864317Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7864526Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7864736Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7864739Z 
2025-07-17T11:07:26.7864828Z SIGABRT(6), PID: 1822840, Thread 1823046: 
2025-07-17T11:07:26.7865292Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7865564Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7865768Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7865967Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7866464Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7866732Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7866946Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7866950Z 
2025-07-17T11:07:26.7867097Z SIGABRT(6), PID: 1822837, Thread 1822955: 
2025-07-17T11:07:26.7867577Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7867791Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7867997Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7868193Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7868701Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7868908Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7869116Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7869122Z 
2025-07-17T11:07:26.7869211Z SIGABRT(6), PID: 1822840, Thread 1823050: 
2025-07-17T11:07:26.7869681Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7869888Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7870090Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7870358Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7870858Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7871069Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7871278Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7871283Z 
2025-07-17T11:07:26.7871374Z SIGABRT(6), PID: 1822837, Thread 1822960: 
2025-07-17T11:07:26.7871842Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7872046Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7872248Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7872450Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7872942Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7873150Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7873358Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7873469Z 
2025-07-17T11:07:26.7873565Z SIGABRT(6), PID: 1822840, Thread 1823054: 
2025-07-17T11:07:26.7874031Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7874251Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7874525Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7874724Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7875218Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7875489Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7875714Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7875718Z 
2025-07-17T11:07:26.7875810Z SIGABRT(6), PID: 1822837, Thread 1822964: 
2025-07-17T11:07:26.7876279Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7876490Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7876694Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7876895Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7877387Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7877596Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7877804Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7877808Z 
2025-07-17T11:07:26.7877899Z SIGABRT(6), PID: 1822840, Thread 1823058: 
2025-07-17T11:07:26.7878360Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7878639Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7878848Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7879048Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7879544Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7879758Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7879965Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7879972Z 
2025-07-17T11:07:26.7880060Z SIGABRT(6), PID: 1822837, Thread 1822967: 
2025-07-17T11:07:26.7880533Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7880740Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7880944Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7881139Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7881636Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7881924Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7882136Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7882140Z 
2025-07-17T11:07:26.7882227Z SIGABRT(6), PID: 1822840, Thread 1823062: 
2025-07-17T11:07:26.7882763Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7882970Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7883174Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7883429Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7883930Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7884141Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7884352Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7884358Z 
2025-07-17T11:07:26.7884447Z SIGABRT(6), PID: 1822837, Thread 1822970: 
2025-07-17T11:07:26.7884913Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7885118Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7885324Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7885518Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7886013Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7886219Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7886471Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7886563Z 
2025-07-17T11:07:26.7886733Z SIGABRT(6), PID: 1822840, Thread 1823065: 
2025-07-17T11:07:26.7887258Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7887509Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7887762Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7888004Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7888596Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7888807Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7889027Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7889031Z 
2025-07-17T11:07:26.7889123Z SIGABRT(6), PID: 1822837, Thread 1822973: 
2025-07-17T11:07:26.7889600Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7889849Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7890182Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7890402Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7890899Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7891179Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7891403Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7891407Z 
2025-07-17T11:07:26.7891500Z SIGABRT(6), PID: 1822840, Thread 1823068: 
2025-07-17T11:07:26.7892025Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7892244Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7892451Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7892654Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7893146Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7893359Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7893567Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7893658Z SIGABRT(6), PID: 1822837, Thread 1822975: 
2025-07-17T11:07:26.7894126Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7894335Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7894534Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7894730Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7895285Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7895505Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7895712Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7895716Z 
2025-07-17T11:07:26.7895720Z 
2025-07-17T11:07:26.7895811Z SIGABRT(6), PID: 1822837, Thread 1822978: 
2025-07-17T11:07:26.7896348Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7896560Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7896763Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7896962Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7897459Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7897669Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7897874Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7897881Z 
2025-07-17T11:07:26.7897971Z SIGABRT(6), PID: 1822840, Thread 1823071: 
2025-07-17T11:07:26.7898529Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7898738Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7898946Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7899224Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7899721Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7899926Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7900208Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7900213Z 
2025-07-17T11:07:26.7900305Z SIGABRT(6), PID: 1822837, Thread 1822982: 
2025-07-17T11:07:26.7900774Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7900977Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7901181Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7901382Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7901879Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7902086Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7902297Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7902303Z 
2025-07-17T11:07:26.7902390Z SIGABRT(6), PID: 1822840, Thread 1823074: 
2025-07-17T11:07:26.7902855Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7903060Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7903368Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7903569Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7904066Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7904275Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7904487Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7904575Z SIGABRT(6), PID: 1822837, Thread 1822985: 
2025-07-17T11:07:26.7905047Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7905252Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7905459Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7905656Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7906157Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7906365Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7906641Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7906645Z 
2025-07-17T11:07:26.7906649Z 
2025-07-17T11:07:26.7906737Z SIGABRT(6), PID: 1822840, Thread 1823077: 
2025-07-17T11:07:26.7907201Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7907475Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7907681Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7907877Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7908447Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7908660Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7908873Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7908877Z 
2025-07-17T11:07:26.7908965Z SIGABRT(6), PID: 1822837, Thread 1822989: 
2025-07-17T11:07:26.7909433Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7909642Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7909842Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7910040Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7910533Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7910741Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7910950Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7910954Z 
2025-07-17T11:07:26.7911044Z SIGABRT(6), PID: 1822840, Thread 1823080: 
2025-07-17T11:07:26.7911578Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7911791Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7911993Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7912196Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7912689Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7912901Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7913109Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7913115Z 
2025-07-17T11:07:26.7913204Z SIGABRT(6), PID: 1822837, Thread 1822993: 
2025-07-17T11:07:26.7913672Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7913880Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7914081Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7914280Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7914838Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7915048Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7915254Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7915317Z 
2025-07-17T11:07:26.7915412Z SIGABRT(6), PID: 1822837, Thread 1822997: 
2025-07-17T11:07:26.7915875Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7916083Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7916343Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7916549Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7917045Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7917252Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7917463Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7917467Z 
2025-07-17T11:07:26.7917561Z SIGABRT(6), PID: 1822840, Thread 1823083: 
2025-07-17T11:07:26.7918024Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7918234Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7918436Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7918636Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7919128Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7919339Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7919628Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7919636Z 
2025-07-17T11:07:26.7919726Z SIGABRT(6), PID: 1822837, Thread 1823001: 
2025-07-17T11:07:26.7920194Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7920400Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7920605Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7920801Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7921307Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7921516Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7921724Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7921728Z 
2025-07-17T11:07:26.7921816Z SIGABRT(6), PID: 1822840, Thread 1823086: 
2025-07-17T11:07:26.7922297Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7922569Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7922783Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7922983Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7923482Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7923753Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7923963Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7923967Z 
2025-07-17T11:07:26.7924056Z SIGABRT(6), PID: 1822837, Thread 1823005: 
2025-07-17T11:07:26.7924587Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7924799Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7925006Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7925204Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7925699Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7925904Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7926113Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7926117Z 
2025-07-17T11:07:26.7926209Z SIGABRT(6), PID: 1822837, Thread 1823009: 
2025-07-17T11:07:26.7926679Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7926885Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7927091Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7927286Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7927853Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7928061Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7928276Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7928280Z 
2025-07-17T11:07:26.7928369Z SIGABRT(6), PID: 1822840, Thread 1823089: 
2025-07-17T11:07:26.7928839Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7929045Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7929251Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7929453Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7929946Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7930153Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7930363Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7930433Z 
2025-07-17T11:07:26.7930526Z SIGABRT(6), PID: 1822837, Thread 1823013: 
2025-07-17T11:07:26.7930994Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7931201Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7931472Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7931675Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7932171Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7932479Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7932693Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7932700Z 
2025-07-17T11:07:26.7932796Z SIGABRT(6), PID: 1822840, Thread 1823092: 
2025-07-17T11:07:26.7933260Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7933469Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7933676Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7933875Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7934367Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7934576Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7934784Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7934787Z 
2025-07-17T11:07:26.7934877Z SIGABRT(6), PID: 1822837, Thread 1823017: 
2025-07-17T11:07:26.7935339Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7935614Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7935821Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7936081Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7936579Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7936790Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7936995Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7937000Z 
2025-07-17T11:07:26.7937095Z SIGABRT(6), PID: 1822840, Thread 1823095: 
2025-07-17T11:07:26.7937562Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7937778Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7937979Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7938181Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7938683Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7938979Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7939186Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7939191Z 
2025-07-17T11:07:26.7939283Z SIGABRT(6), PID: 1822837, Thread 1823021: 
2025-07-17T11:07:26.7939826Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7940035Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7940240Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7940513Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7941011Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7941223Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7941434Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7941438Z 
2025-07-17T11:07:26.7941529Z SIGABRT(6), PID: 1822840, Thread 1823098: 
2025-07-17T11:07:26.7941996Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7942203Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7942409Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7942608Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7943107Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7943312Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7943522Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7943528Z 
2025-07-17T11:07:26.7943692Z SIGABRT(6), PID: 1822837, Thread 1823025: 
2025-07-17T11:07:26.7944164Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7944370Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7944576Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7944776Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7945271Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7945476Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7945690Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7945694Z 
2025-07-17T11:07:26.7945785Z SIGABRT(6), PID: 1822840, Thread 1823101: 
2025-07-17T11:07:26.7946255Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7946464Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7946669Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7946948Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7947451Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7947657Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7947936Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7947941Z 
2025-07-17T11:07:26.7948033Z SIGABRT(6), PID: 1822837, Thread 1823030: 
2025-07-17T11:07:26.7948505Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7948864Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7949074Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7949268Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7949761Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7949973Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7950184Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7950187Z 
2025-07-17T11:07:26.7950276Z SIGABRT(6), PID: 1822840, Thread 1823104: 
2025-07-17T11:07:26.7950749Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7950961Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7951163Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7951365Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7951857Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7952149Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7952361Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7952366Z 
2025-07-17T11:07:26.7952456Z SIGABRT(6), PID: 1822837, Thread 1823034: 
2025-07-17T11:07:26.7952926Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7953137Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7953336Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7953531Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7954024Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7954233Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7954439Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7954443Z 
2025-07-17T11:07:26.7954535Z SIGABRT(6), PID: 1822840, Thread 1823107: 
2025-07-17T11:07:26.7954999Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7955284Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7955487Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7955686Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7956248Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7956458Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7956667Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7956735Z 
2025-07-17T11:07:26.7956830Z SIGABRT(6), PID: 1822837, Thread 1823038: 
2025-07-17T11:07:26.7957299Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7957507Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7957709Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7957915Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7958408Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7958619Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7958826Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7958831Z 
2025-07-17T11:07:26.7958922Z SIGABRT(6), PID: 1822840, Thread 1823110: 
2025-07-17T11:07:26.7959385Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7959593Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7959796Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7960063Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7960563Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7960774Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7960983Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7960988Z 
2025-07-17T11:07:26.7961083Z SIGABRT(6), PID: 1822837, Thread 1823042: 
2025-07-17T11:07:26.7961557Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7961761Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7961969Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7962166Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7962670Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7962878Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7963165Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7963255Z SIGABRT(6), PID: 1822840, Thread 1823113: 
2025-07-17T11:07:26.7963723Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7964019Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7964227Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7964424Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7964985Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7965196Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7965411Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7965415Z 
2025-07-17T11:07:26.7965419Z 
2025-07-17T11:07:26.7965506Z SIGABRT(6), PID: 1822840, Thread 1823117: 
2025-07-17T11:07:26.7965976Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7966187Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7966394Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7966590Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7967087Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7967296Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7967507Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7967511Z 
2025-07-17T11:07:26.7967598Z SIGABRT(6), PID: 1822837, Thread 1823047: 
2025-07-17T11:07:26.7968137Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7968348Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7968552Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7968750Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7969244Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7969455Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7969664Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7969668Z 
2025-07-17T11:07:26.7969756Z SIGABRT(6), PID: 1822840, Thread 1823121: 
2025-07-17T11:07:26.7970234Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7970440Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7970643Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7970840Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7971411Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7971619Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7971828Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7971894Z 
2025-07-17T11:07:26.7971989Z SIGABRT(6), PID: 1822837, Thread 1823051: 
2025-07-17T11:07:26.7972457Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7972666Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7972936Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7973282Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7973807Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7974014Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7974219Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7974227Z 
2025-07-17T11:07:26.7974321Z SIGABRT(6), PID: 1822843, Thread 1822843: 
2025-07-17T11:07:26.7974794Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7975001Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7975237Z frame #2: <unknown function> + 0x77435 (0x7f1b82288435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7975472Z frame #3: <unknown function> + 0x76c8e (0x7f1b82287c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7975695Z frame #4: <unknown function> + 0x6ad69 (0x7f1b8227bd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.7975971Z frame #5: <unknown function> + 0x3f54db (0x7f1c12ee04db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7976186Z frame #6: <unknown function> + 0x3e11d8 (0x7f1c12ecc1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7976482Z frame #7: <unknown function> + 0xb2f86 (0x7f1c12b9df86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7976697Z frame #8: <unknown function> + 0x8e501 (0x7f1c12b79501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.7977087Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f1c3d9807ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.7977475Z frame #10: <unknown function> + 0xd307d9 (0x7f1c5bed37d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7977707Z frame #11: <unknown function> + 0x139902 (0x56103286f902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7977957Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x561032866a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7978186Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7978429Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x561032866a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7978659Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7978896Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7979121Z frame #17: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7979354Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7979660Z frame #19: <unknown function> + 0x1474b2 (0x56103287d4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7979865Z frame #20: PyObject_Call + 0xbc (0x56103287de4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7980086Z frame #21: <unknown function> + 0x1ff69d (0x56103293569d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7980392Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x56103286b023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7980634Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x561032866cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7980855Z frame #24: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7981164Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x5610328623bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7981387Z frame #26: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7981623Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x5610328623bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7981840Z frame #28: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7982081Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x561032866a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7982336Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x56103286a4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7982553Z frame #31: _PyObject_Call_Prepend + 0x69 (0x56103287bab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7982771Z frame #32: <unknown function> + 0x205949 (0x56103293b949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7982993Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x56103286b023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7983231Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x561032866cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7983455Z frame #35: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7983687Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x5610328623bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7983908Z frame #37: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7984206Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x5610328623bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7984433Z frame #39: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7984662Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x5610328623bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7984882Z frame #41: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7985123Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x5610328633f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7985347Z frame #43: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7985736Z frame #44: <unknown function> + 0x7ab3f7 (0x7f1c5b94e3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.7985957Z frame #45: <unknown function> + 0x13bc88 (0x561032871c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7986166Z frame #46: PyObject_Call + 0x20f (0x56103287df9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7986410Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x56103286798d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7986628Z frame #48: <unknown function> + 0x147608 (0x56103287d608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7986869Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x561032864c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7987089Z frame #50: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7987397Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x5610328623bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7987614Z frame #52: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7987855Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x5610328633f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7988143Z frame #54: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7988379Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x5610328623bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7988598Z frame #56: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7988902Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x561032864c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7989125Z frame #58: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7989367Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x561032864c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7989581Z frame #60: <unknown function> + 0x147608 (0x56103287d608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7989818Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x561032864c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7990036Z frame #62: <unknown function> + 0x147608 (0x56103287d608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.7990040Z 
2025-07-17T11:07:26.7990134Z SIGABRT(6), PID: 1822840, Thread 1823124: 
2025-07-17T11:07:26.7990612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7990831Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7991037Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7991245Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7991755Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7991968Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7992250Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7992255Z 
2025-07-17T11:07:26.7992353Z SIGABRT(6), PID: 1822837, Thread 1823055: 
2025-07-17T11:07:26.7992830Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7993039Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7993248Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7993446Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7993945Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7994154Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7994362Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7994366Z 
2025-07-17T11:07:26.7994454Z SIGABRT(6), PID: 1822843, Thread 1823125: 
2025-07-17T11:07:26.7994924Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7995230Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7995442Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7995638Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7996136Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7996407Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7996617Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7996708Z SIGABRT(6), PID: 1822840, Thread 1823129: 
2025-07-17T11:07:26.7997244Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7997456Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7997664Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7997864Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7998366Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.7998574Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7998786Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7998790Z 
2025-07-17T11:07:26.7998794Z 
2025-07-17T11:07:26.7998882Z SIGABRT(6), PID: 1822837, Thread 1823059: 
2025-07-17T11:07:26.7999354Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.7999562Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7999765Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.7999960Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8000522Z frame #4: <unknown function> + 0x3475ab (0x7f0c8bf845ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8000732Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8000941Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8001031Z SIGABRT(6), PID: 1822840, Thread 1823135: 
2025-07-17T11:07:26.8001502Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8001707Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8001914Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8002111Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8002608Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8002813Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8003027Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8003117Z SIGABRT(6), PID: 1822843, Thread 1823133: 
2025-07-17T11:07:26.8003668Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8003879Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8004084Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8004358Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8004861Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8005065Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8005340Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8005347Z 
2025-07-17T11:07:26.8005350Z 
2025-07-17T11:07:26.8005353Z 
2025-07-17T11:07:26.8005447Z SIGABRT(6), PID: 1822837, Thread 1823896: 
2025-07-17T11:07:26.8005916Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8006369Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f0b9d7e208a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8006576Z frame #2: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8006768Z frame #3: pthread_kill + 0x12c (0x7f0c8f0ee9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8006934Z frame #4: raise + 0x16 (0x7f0c8f09a476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8007099Z frame #5: abort + 0xd3 (0x7f0c8f0807f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8007335Z frame #6: <unknown function> + 0x972cc (0x7f0b0b36b2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8007566Z frame #7: <unknown function> + 0x95f47 (0x7f0b0b369f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8007787Z frame #8: <unknown function> + 0x34b71 (0x7f0b0b308b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8007997Z frame #9: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8008278Z frame #10: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8008283Z 
2025-07-17T11:07:26.8008378Z SIGABRT(6), PID: 1822840, Thread 1823140: 
2025-07-17T11:07:26.8008848Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8009064Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8009269Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8009469Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8009966Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8010180Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8010388Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8010392Z 
2025-07-17T11:07:26.8010482Z SIGABRT(6), PID: 1822843, Thread 1823139: 
2025-07-17T11:07:26.8010950Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8011158Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8011444Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8011643Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8012138Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8012422Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8012632Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8012636Z 
2025-07-17T11:07:26.8012726Z SIGABRT(6), PID: 1822837, Thread 1826815: 
2025-07-17T11:07:26.8013260Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8013474Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8013642Z frame #2: __poll + 0x4f (0x7f0c8f170bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8014056Z frame #3: <unknown function> + 0x489f (0x7f0c8ec7c89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.8014282Z frame #4: <unknown function> + 0x1483bc (0x560b1be263bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8014529Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8014756Z frame #6: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8014990Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8015214Z frame #8: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8015461Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x560b1be0ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8015690Z frame #10: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8016007Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x560b1be0cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8016234Z frame #12: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8016564Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8016790Z frame #14: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8017030Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8017250Z frame #16: <unknown function> + 0x147734 (0x560b1be25734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8017481Z frame #17: <unknown function> + 0x21ca82 (0x560b1befaa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8017700Z frame #18: <unknown function> + 0x21c9e4 (0x560b1befa9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8017913Z frame #19: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8018125Z frame #20: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8018132Z 
2025-07-17T11:07:26.8018227Z SIGABRT(6), PID: 1822840, Thread 1823145: 
2025-07-17T11:07:26.8018699Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8018913Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8019123Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8019410Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8019912Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8020127Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8020424Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8020431Z 
2025-07-17T11:07:26.8020520Z SIGABRT(6), PID: 1822843, Thread 1823144: 
2025-07-17T11:07:26.8020993Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8021278Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8021489Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8021691Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8022207Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8022419Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8022631Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8022635Z 
2025-07-17T11:07:26.8022727Z SIGABRT(6), PID: 1822840, Thread 1823150: 
2025-07-17T11:07:26.8023206Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8023414Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8023622Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8023822Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8024322Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8024620Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8024835Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8024839Z 
2025-07-17T11:07:26.8024929Z SIGABRT(6), PID: 1822843, Thread 1823148: 
2025-07-17T11:07:26.8025406Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8025613Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8025816Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8026012Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8026513Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8026719Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8026926Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8026930Z 
2025-07-17T11:07:26.8027021Z SIGABRT(6), PID: 1822840, Thread 1823155: 
2025-07-17T11:07:26.8027502Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8027784Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8027994Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8028190Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8028758Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8028967Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8029176Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8029181Z 
2025-07-17T11:07:26.8029328Z SIGABRT(6), PID: 1822837, Thread 1827351: 
2025-07-17T11:07:26.8029800Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8030008Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8030215Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8030440Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f0c8f0ec2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8030892Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f0bc952a048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8031316Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f0bc952cbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8031553Z frame #6: <unknown function> + 0xdc253 (0x7f0b9ba4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8031772Z frame #7: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8031979Z frame #8: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8031983Z 
2025-07-17T11:07:26.8032074Z SIGABRT(6), PID: 1822840, Thread 1823160: 
2025-07-17T11:07:26.8032542Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8032822Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8033030Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8033230Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8033733Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8033943Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8034153Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8034157Z 
2025-07-17T11:07:26.8034247Z SIGABRT(6), PID: 1822843, Thread 1823153: 
2025-07-17T11:07:26.8034716Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8034925Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8035126Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8035327Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8035824Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8036104Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8036313Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8036317Z 
2025-07-17T11:07:26.8036475Z SIGABRT(6), PID: 1822841, Thread 1822841: 
2025-07-17T11:07:26.8036944Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8037152Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8037381Z frame #2: <unknown function> + 0x77435 (0x7f7f30672435 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8037675Z frame #3: <unknown function> + 0x76c8e (0x7f7f30671c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8037909Z frame #4: <unknown function> + 0x6ad69 (0x7f7f30665d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8038130Z frame #5: <unknown function> + 0x3f54db (0x7f7fc12ca4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8038344Z frame #6: <unknown function> + 0x3e11d8 (0x7f7fc12b61d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8038556Z frame #7: <unknown function> + 0xb2f86 (0x7f7fc0f87f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8038762Z frame #8: <unknown function> + 0x8e501 (0x7f7fc0f63501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8039156Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f7febd6a7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.8039544Z frame #10: <unknown function> + 0xd307d9 (0x7f800a2bd7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8039773Z frame #11: <unknown function> + 0x139902 (0x555c0fca9902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8040025Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x555c0fca0a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8040254Z frame #13: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8040495Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x555c0fca0a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8040798Z frame #15: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8041038Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8041268Z frame #17: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8041504Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8041724Z frame #19: <unknown function> + 0x1474b2 (0x555c0fcb74b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8041928Z frame #20: PyObject_Call + 0xbc (0x555c0fcb7e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8042151Z frame #21: <unknown function> + 0x1ff69d (0x555c0fd6f69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8042371Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x555c0fca5023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8042616Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x555c0fca0cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8042839Z frame #24: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8043077Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x555c0fc9c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8043302Z frame #26: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8043535Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x555c0fc9c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8043828Z frame #28: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8044069Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x555c0fca0a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8044326Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x555c0fca44b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8044619Z frame #31: _PyObject_Call_Prepend + 0x69 (0x555c0fcb5ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8044839Z frame #32: <unknown function> + 0x205949 (0x555c0fd75949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8045065Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x555c0fca5023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8045304Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x555c0fca0cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8045594Z frame #35: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8045835Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x555c0fc9c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8046059Z frame #37: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8046295Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x555c0fc9c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8046524Z frame #39: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8046755Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x555c0fc9c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8046981Z frame #41: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8047218Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x555c0fc9d3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8047445Z frame #43: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8047833Z frame #44: <unknown function> + 0x7ab3f7 (0x7f8009d383f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8048054Z frame #45: <unknown function> + 0x13bc88 (0x555c0fcabc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8048258Z frame #46: PyObject_Call + 0x20f (0x555c0fcb7f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8048568Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x555c0fca198d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8048789Z frame #48: <unknown function> + 0x147608 (0x555c0fcb7608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8049032Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x555c0fc9ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8049257Z frame #50: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8049494Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x555c0fc9c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8049719Z frame #52: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8049956Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x555c0fc9d3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8050175Z frame #54: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8050411Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x555c0fc9c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8050631Z frame #56: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8050874Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x555c0fc9ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8051097Z frame #58: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8051341Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x555c0fc9ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8051632Z frame #60: <unknown function> + 0x147608 (0x555c0fcb7608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8051870Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x555c0fc9ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8052087Z frame #62: <unknown function> + 0x147608 (0x555c0fcb7608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8052154Z 
2025-07-17T11:07:26.8052254Z SIGABRT(6), PID: 1822840, Thread 1823164: 
2025-07-17T11:07:26.8052731Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8052946Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8053238Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8053451Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8053955Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8054194Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8054412Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8054419Z 
2025-07-17T11:07:26.8054518Z SIGABRT(6), PID: 1822843, Thread 1823158: 
2025-07-17T11:07:26.8054993Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8055204Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8055409Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8055608Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8056166Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8056379Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8056673Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8056771Z SIGABRT(6), PID: 1822837, Thread 1827352: 
2025-07-17T11:07:26.8057242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8057451Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8057653Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8057879Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f0c8f0ec2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8058357Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f0bc9524741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8058586Z frame #5: <unknown function> + 0xdc253 (0x7f0b9ba4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8058796Z frame #6: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8059005Z frame #7: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8059094Z SIGABRT(6), PID: 1822841, Thread 1823295: 
2025-07-17T11:07:26.8059565Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8059862Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8060065Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8060265Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8060767Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8061057Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8061265Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8061358Z SIGABRT(6), PID: 1822840, Thread 1823168: 
2025-07-17T11:07:26.8061900Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8062124Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8062329Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8062530Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8063024Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8063239Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8063449Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8063453Z 
2025-07-17T11:07:26.8063456Z 
2025-07-17T11:07:26.8063460Z 
2025-07-17T11:07:26.8063464Z 
2025-07-17T11:07:26.8063558Z SIGABRT(6), PID: 1822837, Thread 1827353: 
2025-07-17T11:07:26.8064028Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8064235Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8064401Z frame #2: ioctl + 0x3f (0x7f0c8f17294f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8064708Z frame #3: <unknown function> + 0x13eef0 (0x7f0b0b412ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8064943Z frame #4: <unknown function> + 0x1378c4 (0x7f0b0b40b8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8065177Z frame #5: <unknown function> + 0x95b2b (0x7f0b0b369b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8065401Z frame #6: <unknown function> + 0x34b71 (0x7f0b0b308b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8065613Z frame #7: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8065822Z frame #8: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8065915Z SIGABRT(6), PID: 1822840, Thread 1823172: 
2025-07-17T11:07:26.8066383Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8066595Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8066802Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8067008Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8067507Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8067722Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8068003Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8068096Z SIGABRT(6), PID: 1822841, Thread 1823296: 
2025-07-17T11:07:26.8068567Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8068845Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8069049Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8069254Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8069816Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8070037Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8070248Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8070256Z 
2025-07-17T11:07:26.8070345Z SIGABRT(6), PID: 1822843, Thread 1823163: 
2025-07-17T11:07:26.8070816Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8071026Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8071232Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8071425Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8071929Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8072137Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8072345Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8072349Z 
2025-07-17T11:07:26.8072352Z 
2025-07-17T11:07:26.8072356Z 
2025-07-17T11:07:26.8072445Z SIGABRT(6), PID: 1822843, Thread 1823167: 
2025-07-17T11:07:26.8072989Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8073199Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8073404Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8073600Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8074095Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8074299Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8074511Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8074517Z 
2025-07-17T11:07:26.8074607Z SIGABRT(6), PID: 1822841, Thread 1823297: 
2025-07-17T11:07:26.8075078Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8075281Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8075486Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8075680Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8076253Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8076458Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8076665Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8076836Z 
2025-07-17T11:07:26.8076927Z SIGABRT(6), PID: 1822840, Thread 1823176: 
2025-07-17T11:07:26.8077400Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8077608Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8077875Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8078080Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8078574Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8078783Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8078997Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8079001Z 
2025-07-17T11:07:26.8079091Z SIGABRT(6), PID: 1822837, Thread 1827653: 
2025-07-17T11:07:26.8079562Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8079776Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8079942Z frame #2: __poll + 0x4f (0x7f0c8f170bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8080153Z frame #3: <unknown function> + 0xd4540 (0x7f0b53582540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8080356Z frame #4: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8080563Z frame #5: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8080571Z 
2025-07-17T11:07:26.8080724Z SIGABRT(6), PID: 1822837, Thread 1827660: 
2025-07-17T11:07:26.8081196Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8081400Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8081565Z frame #2: __poll + 0x4f (0x7f0c8f170bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8081778Z frame #3: <unknown function> + 0xd65a2 (0x7f0b535845a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8081988Z frame #4: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8082197Z frame #5: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8082288Z SIGABRT(6), PID: 1822843, Thread 1823171: 
2025-07-17T11:07:26.8082754Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8082964Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8083166Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8083369Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8083863Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8084137Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8084346Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8084350Z 
2025-07-17T11:07:26.8084353Z 
2025-07-17T11:07:26.8084528Z SIGABRT(6), PID: 1822841, Thread 1823298: 
2025-07-17T11:07:26.8084998Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8085208Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8085411Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8085668Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8086169Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8086376Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8086582Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8086588Z 
2025-07-17T11:07:26.8086680Z SIGABRT(6), PID: 1822840, Thread 1823179: 
2025-07-17T11:07:26.8087147Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8093387Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8093663Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8093893Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8094422Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8094648Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8094982Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8094988Z 
2025-07-17T11:07:26.8095096Z SIGABRT(6), PID: 1822837, Thread 1827667: 
2025-07-17T11:07:26.8095593Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8095814Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8096095Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8096304Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8096524Z frame #4: <unknown function> + 0xd1e7f (0x7f0b5357fe7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8096741Z frame #5: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8096959Z frame #6: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8096964Z 
2025-07-17T11:07:26.8097062Z SIGABRT(6), PID: 1822843, Thread 1823175: 
2025-07-17T11:07:26.8097554Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8097776Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8097980Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8098289Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8098809Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8099022Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8099316Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8099323Z 
2025-07-17T11:07:26.8099417Z SIGABRT(6), PID: 1822841, Thread 1823299: 
2025-07-17T11:07:26.8099895Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8100172Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8100383Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8100583Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8101086Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8101295Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8101505Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8101509Z 
2025-07-17T11:07:26.8101597Z SIGABRT(6), PID: 1822840, Thread 1823184: 
2025-07-17T11:07:26.8102072Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8102287Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8102495Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8102692Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8103190Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8103478Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8103694Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8103698Z 
2025-07-17T11:07:26.8103787Z SIGABRT(6), PID: 1822837, Thread 1827670: 
2025-07-17T11:07:26.8104258Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8104465Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8104670Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8104876Z frame #3: <unknown function> + 0x9ca53 (0x7f0c8f0f4a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8105138Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x560b1bdf2bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8105385Z frame #5: <unknown function> + 0x1d314f (0x560b1beb114f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8105614Z frame #6: <unknown function> + 0x1d2f6b (0x560b1beb0f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8105838Z frame #7: <unknown function> + 0x6df36 (0x560b1bd4bf36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8106088Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8106383Z frame #9: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8106631Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8106859Z frame #11: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8107098Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8107388Z frame #13: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8107625Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8107850Z frame #15: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8108149Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8108377Z frame #17: <unknown function> + 0x147734 (0x560b1be25734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8108599Z frame #18: <unknown function> + 0x21ca82 (0x560b1befaa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8108816Z frame #19: <unknown function> + 0x21c9e4 (0x560b1befa9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8109039Z frame #20: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8109257Z frame #21: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8109262Z 
2025-07-17T11:07:26.8109356Z SIGABRT(6), PID: 1822843, Thread 1823178: 
2025-07-17T11:07:26.8109841Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8110060Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8110267Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8110469Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8110977Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8111253Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8111472Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8111476Z 
2025-07-17T11:07:26.8111570Z SIGABRT(6), PID: 1822841, Thread 1823300: 
2025-07-17T11:07:26.8112044Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8112252Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8112454Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8112655Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8113153Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8113364Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8113569Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8113662Z SIGABRT(6), PID: 1822840, Thread 1823188: 
2025-07-17T11:07:26.8114130Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8114415Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8114621Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8114823Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8115315Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8115592Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8115808Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8115897Z SIGABRT(6), PID: 1822843, Thread 1823182: 
2025-07-17T11:07:26.8116430Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8116643Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8116848Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8117045Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8117544Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8117748Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8117959Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8117964Z 
2025-07-17T11:07:26.8117967Z 
2025-07-17T11:07:26.8117972Z 
2025-07-17T11:07:26.8118061Z SIGABRT(6), PID: 1822837, Thread 1827679: 
2025-07-17T11:07:26.8118533Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8118739Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8118904Z frame #2: read + 0x4c (0x7f0c8f16c81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8119086Z frame #3: _Py_read + 0x4c (0x560b1beb8a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8119381Z frame #4: <unknown function> + 0x208c3b (0x560b1bee6c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8119605Z frame #5: <unknown function> + 0x14648d (0x560b1be2448d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8119852Z frame #6: PyObject_VectorcallMethod + 0x85 (0x560b1be33215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8120076Z frame #7: <unknown function> + 0x208ae7 (0x560b1bee6ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8120298Z frame #8: <unknown function> + 0x208472 (0x560b1bee6472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8120517Z frame #9: <unknown function> + 0x1da6ee (0x560b1beb86ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8120744Z frame #10: <unknown function> + 0x1483bc (0x560b1be263bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8120987Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8121215Z frame #12: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8121452Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x560b1be0a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8121671Z frame #14: <unknown function> + 0x147734 (0x560b1be25734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8121932Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x560b1be0cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8122250Z frame #16: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8122483Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8122708Z frame #18: _PyFunction_Vectorcall + 0x6c (0x560b1be1a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8122939Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x560b1be0a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8123238Z frame #20: <unknown function> + 0x147734 (0x560b1be25734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8123458Z frame #21: <unknown function> + 0x21ca82 (0x560b1befaa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8123675Z frame #22: <unknown function> + 0x21c9e4 (0x560b1befa9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8123947Z frame #23: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8124169Z frame #24: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8124176Z 
2025-07-17T11:07:26.8124269Z SIGABRT(6), PID: 1822841, Thread 1823301: 
2025-07-17T11:07:26.8124746Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8124955Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8125165Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8125362Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8125862Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8126068Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8126278Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8126283Z 
2025-07-17T11:07:26.8126370Z SIGABRT(6), PID: 1822840, Thread 1823192: 
2025-07-17T11:07:26.8126839Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8127115Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8127324Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8127528Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8128028Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8128240Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8128451Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8128542Z SIGABRT(6), PID: 1822843, Thread 1823186: 
2025-07-17T11:07:26.8129014Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8129225Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8129431Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8129627Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8130124Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8130402Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8130615Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8130619Z 
2025-07-17T11:07:26.8130623Z 
2025-07-17T11:07:26.8130715Z SIGABRT(6), PID: 1822843, Thread 1823190: 
2025-07-17T11:07:26.8131257Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8131467Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8131668Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8131927Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8132427Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8132638Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8132846Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8132850Z 
2025-07-17T11:07:26.8132942Z SIGABRT(6), PID: 1822841, Thread 1823302: 
2025-07-17T11:07:26.8133407Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8133614Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8133816Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8134014Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8134508Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8134715Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8134923Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8134929Z 
2025-07-17T11:07:26.8135096Z SIGABRT(6), PID: 1822840, Thread 1823195: 
2025-07-17T11:07:26.8135564Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8135774Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8136045Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8136247Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8136740Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8136951Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8137161Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8137166Z 
2025-07-17T11:07:26.8137257Z SIGABRT(6), PID: 1822843, Thread 1823194: 
2025-07-17T11:07:26.8137718Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8137925Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8138126Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8138410Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8138900Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8139107Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8139394Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8139399Z 
2025-07-17T11:07:26.8139491Z SIGABRT(6), PID: 1822841, Thread 1823303: 
2025-07-17T11:07:26.8139955Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8140232Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8140442Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8140642Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8141140Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8141350Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8141557Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8141564Z 
2025-07-17T11:07:26.8141654Z SIGABRT(6), PID: 1822840, Thread 1823198: 
2025-07-17T11:07:26.8142130Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8142345Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8142552Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8142748Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8143320Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8143535Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8143746Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8143750Z 
2025-07-17T11:07:26.8143838Z SIGABRT(6), PID: 1822843, Thread 1823197: 
2025-07-17T11:07:26.8144311Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8144518Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8144724Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8144914Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8145410Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8145614Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8145821Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8145909Z SIGABRT(6), PID: 1822841, Thread 1823304: 
2025-07-17T11:07:26.8146380Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8146655Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8146860Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8147058Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8147619Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8147822Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8148029Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8148033Z 
2025-07-17T11:07:26.8148096Z 
2025-07-17T11:07:26.8148189Z SIGABRT(6), PID: 1822840, Thread 1823201: 
2025-07-17T11:07:26.8148665Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8148870Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8149077Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8149276Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8149771Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8149976Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8150190Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8150195Z 
2025-07-17T11:07:26.8150283Z SIGABRT(6), PID: 1822843, Thread 1823200: 
2025-07-17T11:07:26.8150751Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8150954Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8151159Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8151439Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8151943Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8152151Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8152357Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8152449Z SIGABRT(6), PID: 1822841, Thread 1823305: 
2025-07-17T11:07:26.8152914Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8153119Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8153324Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8153517Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8154006Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8154214Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8154489Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8154494Z 
2025-07-17T11:07:26.8154497Z 
2025-07-17T11:07:26.8154587Z SIGABRT(6), PID: 1822840, Thread 1823204: 
2025-07-17T11:07:26.8155050Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8155322Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8155524Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8155722Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8156270Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8156484Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8156694Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8156700Z 
2025-07-17T11:07:26.8156791Z SIGABRT(6), PID: 1822843, Thread 1823203: 
2025-07-17T11:07:26.8157259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8157470Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8157674Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8157872Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8158363Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8158571Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8158776Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8158780Z 
2025-07-17T11:07:26.8158871Z SIGABRT(6), PID: 1822841, Thread 1823306: 
2025-07-17T11:07:26.8159401Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8159615Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8159816Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8160012Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8160503Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8160711Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8160918Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8161010Z SIGABRT(6), PID: 1822840, Thread 1823207: 
2025-07-17T11:07:26.8161477Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8161687Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8161891Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8162092Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8162651Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8162866Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8163072Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8163139Z 
2025-07-17T11:07:26.8163143Z 
2025-07-17T11:07:26.8163235Z SIGABRT(6), PID: 1822843, Thread 1823206: 
2025-07-17T11:07:26.8163705Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8163909Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8164171Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8164368Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8164867Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8165069Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8165279Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8165285Z 
2025-07-17T11:07:26.8165371Z SIGABRT(6), PID: 1822837, Thread 1827694: 
2025-07-17T11:07:26.8165841Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8166049Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8166256Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8166454Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8166831Z frame #4: <unknown function> + 0x5a02023 (0x7f0bd3518023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8167461Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f0bd351c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8168059Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f0bd35131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8168448Z frame #7: <unknown function> + 0x79d655 (0x7f0be4a03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8168691Z frame #8: <unknown function> + 0xdc253 (0x7f0b9ba4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8168906Z frame #9: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8169123Z frame #10: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8169128Z 
2025-07-17T11:07:26.8169223Z SIGABRT(6), PID: 1822840, Thread 1823210: 
2025-07-17T11:07:26.8169709Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8169922Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8170128Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8170332Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8170836Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8171118Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8171333Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8171337Z 
2025-07-17T11:07:26.8171489Z SIGABRT(6), PID: 1822841, Thread 1823307: 
2025-07-17T11:07:26.8171967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8172173Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8172375Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8172631Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8173137Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8173344Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8173552Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8173559Z 
2025-07-17T11:07:26.8173653Z SIGABRT(6), PID: 1822843, Thread 1823209: 
2025-07-17T11:07:26.8174122Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8174329Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8174536Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8174736Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8175239Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8175447Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8175725Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8175730Z 
2025-07-17T11:07:26.8175827Z SIGABRT(6), PID: 1822840, Thread 1823213: 
2025-07-17T11:07:26.8176370Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8176583Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8176788Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8176990Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8177487Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8177699Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8177911Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8177915Z 
2025-07-17T11:07:26.8178007Z SIGABRT(6), PID: 1822843, Thread 1823212: 
2025-07-17T11:07:26.8178475Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8178687Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8178984Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8179185Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8179677Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8179963Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8180171Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8180175Z 
2025-07-17T11:07:26.8180267Z SIGABRT(6), PID: 1822841, Thread 1823308: 
2025-07-17T11:07:26.8180828Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8181043Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8181248Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8181446Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8181941Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8182153Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8182368Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8182375Z 
2025-07-17T11:07:26.8182464Z SIGABRT(6), PID: 1822843, Thread 1823215: 
2025-07-17T11:07:26.8182937Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8183141Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8183343Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8183538Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8184116Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8184325Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8184538Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8184542Z 
2025-07-17T11:07:26.8184632Z SIGABRT(6), PID: 1822841, Thread 1823309: 
2025-07-17T11:07:26.8185106Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8185312Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8185520Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8185715Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8186211Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8186415Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8186622Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8186628Z 
2025-07-17T11:07:26.8186715Z SIGABRT(6), PID: 1822840, Thread 1823216: 
2025-07-17T11:07:26.8187250Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8187459Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8187665Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8187932Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8188429Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8188637Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8188908Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8188916Z 
2025-07-17T11:07:26.8189010Z SIGABRT(6), PID: 1822840, Thread 1823219: 
2025-07-17T11:07:26.8189481Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8189689Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8189896Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8190094Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8190594Z frame #4: <unknown function> + 0x3475ab (0x7fa3c7c945ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8190804Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8191013Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8191019Z 
2025-07-17T11:07:26.8191107Z SIGABRT(6), PID: 1822843, Thread 1823218: 
2025-07-17T11:07:26.8191575Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8191779Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8192057Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8192255Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8192751Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8192959Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8193170Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8193174Z 
2025-07-17T11:07:26.8193260Z SIGABRT(6), PID: 1822841, Thread 1823310: 
2025-07-17T11:07:26.8193728Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8193939Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8194140Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8194333Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8194825Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8195108Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8195315Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8195319Z 
2025-07-17T11:07:26.8195411Z SIGABRT(6), PID: 1822840, Thread 1823900: 
2025-07-17T11:07:26.8195890Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8196415Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fa2d94f408a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8196626Z frame #2: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8196826Z frame #3: pthread_kill + 0x12c (0x7fa3cadfe9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8197053Z frame #4: raise + 0x16 (0x7fa3cadaa476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8197221Z frame #5: abort + 0xd3 (0x7fa3cad907f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8197464Z frame #6: <unknown function> + 0x972cc (0x7fa24707d2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8197696Z frame #7: <unknown function> + 0x95f47 (0x7fa24707bf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8197916Z frame #8: <unknown function> + 0x34b71 (0x7fa24701ab71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8198131Z frame #9: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8198348Z frame #10: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8198353Z 
2025-07-17T11:07:26.8198446Z SIGABRT(6), PID: 1822843, Thread 1823221: 
2025-07-17T11:07:26.8198923Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8199136Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8199338Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8199540Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8200102Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8200318Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8200530Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8200534Z 
2025-07-17T11:07:26.8200626Z SIGABRT(6), PID: 1822841, Thread 1823311: 
2025-07-17T11:07:26.8201099Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8201308Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8201510Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8201711Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8202210Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8202421Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8202625Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8202630Z 
2025-07-17T11:07:26.8202722Z SIGABRT(6), PID: 1822840, Thread 1827630: 
2025-07-17T11:07:26.8203188Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8203468Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8203641Z frame #2: __poll + 0x4f (0x7fa3cae80bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8204058Z frame #3: <unknown function> + 0x489f (0x7fa3ca98c89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.8204350Z frame #4: <unknown function> + 0x1483bc (0x5557b749a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8204599Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8204883Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8205125Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8205349Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8205595Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x5557b7482a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8205825Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8206079Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x5557b7480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8206302Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8206539Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8206763Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8207002Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8207219Z frame #16: <unknown function> + 0x147734 (0x5557b7499734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8207438Z frame #17: <unknown function> + 0x21ca82 (0x5557b756ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8207654Z frame #18: <unknown function> + 0x21c9e4 (0x5557b756e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8207871Z frame #19: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8208152Z frame #20: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8208157Z 
2025-07-17T11:07:26.8208258Z SIGABRT(6), PID: 1822843, Thread 1823223: 
2025-07-17T11:07:26.8208744Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8208958Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8209164Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8209370Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8209871Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8210086Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8210291Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8210298Z 
2025-07-17T11:07:26.8210387Z SIGABRT(6), PID: 1822841, Thread 1823312: 
2025-07-17T11:07:26.8210863Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8211245Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8211453Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8211650Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8212150Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8212448Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8212657Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8212664Z 
2025-07-17T11:07:26.8212751Z SIGABRT(6), PID: 1822837, Thread 1827695: 
2025-07-17T11:07:26.8213285Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8213496Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8213703Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8213900Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8214284Z frame #4: <unknown function> + 0x5a02023 (0x7f0bd3518023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8214852Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f0bd351c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8215450Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f0bd35131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8215835Z frame #7: <unknown function> + 0x79d655 (0x7f0be4a03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8216287Z frame #8: <unknown function> + 0xdc253 (0x7f0b9ba4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8216503Z frame #9: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8216816Z frame #10: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8216822Z 
2025-07-17T11:07:26.8216916Z SIGABRT(6), PID: 1822843, Thread 1823225: 
2025-07-17T11:07:26.8217394Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8217602Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8217810Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8218007Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8218508Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8218717Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8218927Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8218931Z 
2025-07-17T11:07:26.8219020Z SIGABRT(6), PID: 1822841, Thread 1823313: 
2025-07-17T11:07:26.8219495Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8219699Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8219991Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8220187Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8220684Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8220975Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8221182Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8221186Z 
2025-07-17T11:07:26.8221278Z SIGABRT(6), PID: 1822843, Thread 1823227: 
2025-07-17T11:07:26.8221821Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8222033Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8222239Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8222444Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8222939Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8223149Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8223357Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8223361Z 
2025-07-17T11:07:26.8223452Z SIGABRT(6), PID: 1822841, Thread 1823314: 
2025-07-17T11:07:26.8223917Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8224125Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8224328Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8224527Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8225085Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8225297Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8225505Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8225510Z 
2025-07-17T11:07:26.8225602Z SIGABRT(6), PID: 1822843, Thread 1823229: 
2025-07-17T11:07:26.8226076Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8226286Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8226488Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8226685Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8227181Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8227389Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8227597Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8227601Z 
2025-07-17T11:07:26.8227762Z SIGABRT(6), PID: 1822840, Thread 1827631: 
2025-07-17T11:07:26.8228232Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8228442Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8228646Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8228941Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa3cadfc2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8229385Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fa30523c048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8229871Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fa30523ebf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8230110Z frame #6: <unknown function> + 0xdc253 (0x7fa2d7760253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8230323Z frame #7: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8230532Z frame #8: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8230541Z 
2025-07-17T11:07:26.8230632Z SIGABRT(6), PID: 1822841, Thread 1823315: 
2025-07-17T11:07:26.8231111Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8231317Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8231521Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8231720Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8232230Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8232437Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8232646Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8232651Z 
2025-07-17T11:07:26.8232739Z SIGABRT(6), PID: 1822843, Thread 1823231: 
2025-07-17T11:07:26.8233276Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8233489Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8233694Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8233891Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8234393Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8234597Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8234804Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8234810Z 
2025-07-17T11:07:26.8234899Z SIGABRT(6), PID: 1822841, Thread 1823316: 
2025-07-17T11:07:26.8235376Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8235580Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8235787Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8236058Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8236556Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8236760Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8237036Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8237040Z 
2025-07-17T11:07:26.8237129Z SIGABRT(6), PID: 1822840, Thread 1827632: 
2025-07-17T11:07:26.8237610Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8237885Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8238096Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8238321Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa3cadfc2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8238795Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fa305236741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8239026Z frame #5: <unknown function> + 0xdc253 (0x7fa2d7760253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8239239Z frame #6: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8239447Z frame #7: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8239452Z 
2025-07-17T11:07:26.8239545Z SIGABRT(6), PID: 1822843, Thread 1823232: 
2025-07-17T11:07:26.8240012Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8240222Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8240423Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8240620Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8241181Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8241398Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8241607Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8241614Z 
2025-07-17T11:07:26.8241704Z SIGABRT(6), PID: 1822840, Thread 1827635: 
2025-07-17T11:07:26.8242177Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8242387Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8242555Z frame #2: ioctl + 0x3f (0x7fa3cae8294f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8242789Z frame #3: <unknown function> + 0x13eef0 (0x7fa247124ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8243024Z frame #4: <unknown function> + 0x1378c4 (0x7fa24711d8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8243253Z frame #5: <unknown function> + 0x95b2b (0x7fa24707bb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8243480Z frame #6: <unknown function> + 0x34b71 (0x7fa24701ab71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8243693Z frame #7: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8243905Z frame #8: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8244094Z SIGABRT(6), PID: 1822841, Thread 1823317: 
2025-07-17T11:07:26.8244569Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8244774Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8245054Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8245251Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8245748Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8246019Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8246235Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8246324Z SIGABRT(6), PID: 1822843, Thread 1823234: 
2025-07-17T11:07:26.8246800Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8247008Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8247215Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8247409Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8247906Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8248111Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8248322Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8248326Z 
2025-07-17T11:07:26.8248329Z 
2025-07-17T11:07:26.8248332Z 
2025-07-17T11:07:26.8248420Z SIGABRT(6), PID: 1822837, Thread 1827696: 
2025-07-17T11:07:26.8248891Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8249164Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8249382Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8249583Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8249962Z frame #4: <unknown function> + 0x5a02023 (0x7f0bd3518023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8250528Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f0bd351c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8251121Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f0bd35131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8251510Z frame #7: <unknown function> + 0x79d655 (0x7f0be4a03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8251744Z frame #8: <unknown function> + 0xdc253 (0x7f0b9ba4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8251953Z frame #9: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8252169Z frame #10: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8252239Z 
2025-07-17T11:07:26.8252329Z SIGABRT(6), PID: 1822843, Thread 1823236: 
2025-07-17T11:07:26.8252802Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8253010Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8253281Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8253476Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8253976Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8254250Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8254462Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8254468Z 
2025-07-17T11:07:26.8254561Z SIGABRT(6), PID: 1822841, Thread 1823318: 
2025-07-17T11:07:26.8255031Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8255238Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8255443Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8255644Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8256224Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8256436Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8256644Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8256648Z 
2025-07-17T11:07:26.8256742Z SIGABRT(6), PID: 1822840, Thread 1827648: 
2025-07-17T11:07:26.8257208Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8257507Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8257680Z frame #2: __poll + 0x4f (0x7fa3cae80bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8257897Z frame #3: <unknown function> + 0xd4540 (0x7fa28f294540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8258103Z frame #4: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8258315Z frame #5: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8258321Z 
2025-07-17T11:07:26.8258410Z SIGABRT(6), PID: 1822843, Thread 1823239: 
2025-07-17T11:07:26.8258882Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8259087Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8259294Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8259488Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8259984Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8260192Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8260485Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8260489Z 
2025-07-17T11:07:26.8260577Z SIGABRT(6), PID: 1822840, Thread 1827655: 
2025-07-17T11:07:26.8261049Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8261255Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8261504Z frame #2: __poll + 0x4f (0x7fa3cae80bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8261718Z frame #3: <unknown function> + 0xd65a2 (0x7fa28f2965a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8261929Z frame #4: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8262214Z frame #5: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8262218Z 
2025-07-17T11:07:26.8262315Z SIGABRT(6), PID: 1822841, Thread 1823319: 
2025-07-17T11:07:26.8262796Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8263005Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8263208Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8263410Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8263911Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8264120Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8264331Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8264340Z 
2025-07-17T11:07:26.8264429Z SIGABRT(6), PID: 1822840, Thread 1827662: 
2025-07-17T11:07:26.8264900Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8265105Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8265379Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8265584Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8265803Z frame #4: <unknown function> + 0xd1e7f (0x7fa28f291e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8266013Z frame #5: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8266226Z frame #6: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8266316Z SIGABRT(6), PID: 1822843, Thread 1823241: 
2025-07-17T11:07:26.8266786Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8266992Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8267201Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8267399Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8267901Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8268107Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8268318Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8268478Z SIGABRT(6), PID: 1822841, Thread 1823320: 
2025-07-17T11:07:26.8268951Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8269156Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8269429Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8269624Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8270121Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8270399Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8270613Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8270617Z 
2025-07-17T11:07:26.8270620Z 
2025-07-17T11:07:26.8270623Z 
2025-07-17T11:07:26.8270713Z SIGABRT(6), PID: 1822841, Thread 1823321: 
2025-07-17T11:07:26.8271186Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8271395Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8271600Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8271796Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8272301Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8272506Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8272713Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8272717Z 
2025-07-17T11:07:26.8272804Z SIGABRT(6), PID: 1822843, Thread 1823243: 
2025-07-17T11:07:26.8273358Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8273572Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8273778Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8273972Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8274469Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8274679Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8274889Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8274979Z SIGABRT(6), PID: 1822840, Thread 1827673: 
2025-07-17T11:07:26.8275455Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8275662Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8275872Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8276079Z frame #3: <unknown function> + 0x9ca53 (0x7fa3cae04a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8276337Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x5557b7466bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8276652Z frame #5: <unknown function> + 0x1d314f (0x5557b752514f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8276877Z frame #6: <unknown function> + 0x1d2f6b (0x5557b7524f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8277100Z frame #7: <unknown function> + 0x6df36 (0x5557b73bff36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8277414Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8277641Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8277883Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8278109Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8278411Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8278642Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8278877Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8279099Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8279338Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8279556Z frame #17: <unknown function> + 0x147734 (0x5557b7499734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8279778Z frame #18: <unknown function> + 0x21ca82 (0x5557b756ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8279994Z frame #19: <unknown function> + 0x21c9e4 (0x5557b756e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8280211Z frame #20: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8280424Z frame #21: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8280428Z 
2025-07-17T11:07:26.8280434Z 
2025-07-17T11:07:26.8280524Z SIGABRT(6), PID: 1822841, Thread 1823322: 
2025-07-17T11:07:26.8280995Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8281275Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8281488Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8281693Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8282193Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8282410Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8282619Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8282623Z 
2025-07-17T11:07:26.8282711Z SIGABRT(6), PID: 1822843, Thread 1823245: 
2025-07-17T11:07:26.8283184Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8283392Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8283596Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8283790Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8284287Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8284566Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8284779Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8284783Z 
2025-07-17T11:07:26.8284872Z SIGABRT(6), PID: 1822837, Thread 1827697: 
2025-07-17T11:07:26.8285411Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8285619Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8285825Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8286087Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8286472Z frame #4: <unknown function> + 0x5a02023 (0x7f0bd3518023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8287042Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f0bd351c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8287638Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f0bd35131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8288028Z frame #7: <unknown function> + 0x79d655 (0x7f0be4a03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8288263Z frame #8: <unknown function> + 0xdc253 (0x7f0b9ba4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8288473Z frame #9: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8288693Z frame #10: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8288782Z SIGABRT(6), PID: 1822840, Thread 1827683: 
2025-07-17T11:07:26.8289257Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8289467Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8289703Z frame #2: read + 0x4c (0x7fa3cae7c81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8289887Z frame #3: _Py_read + 0x4c (0x5557b752ca6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8290116Z frame #4: <unknown function> + 0x208c3b (0x5557b755ac3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8290337Z frame #5: <unknown function> + 0x14648d (0x5557b749848d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8290584Z frame #6: PyObject_VectorcallMethod + 0x85 (0x5557b74a7215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8290804Z frame #7: <unknown function> + 0x208ae7 (0x5557b755aae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8291019Z frame #8: <unknown function> + 0x208472 (0x5557b755a472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8291237Z frame #9: <unknown function> + 0x1da6ee (0x5557b752c6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8291466Z frame #10: <unknown function> + 0x1483bc (0x5557b749a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8291708Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8291933Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8292170Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5557b747e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8292387Z frame #14: <unknown function> + 0x147734 (0x5557b7499734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8292708Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x5557b7480c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8292933Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8293164Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8293468Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5557b748e0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8293700Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5557b747e7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8293917Z frame #20: <unknown function> + 0x147734 (0x5557b7499734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8294137Z frame #21: <unknown function> + 0x21ca82 (0x5557b756ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8294422Z frame #22: <unknown function> + 0x21c9e4 (0x5557b756e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8294645Z frame #23: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8294861Z frame #24: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8294866Z 
2025-07-17T11:07:26.8294869Z 
2025-07-17T11:07:26.8294959Z SIGABRT(6), PID: 1822841, Thread 1823323: 
2025-07-17T11:07:26.8295440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8295650Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8295858Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8296137Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8296641Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8296852Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8297062Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8297066Z 
2025-07-17T11:07:26.8297160Z SIGABRT(6), PID: 1822843, Thread 1823247: 
2025-07-17T11:07:26.8297713Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8297927Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8298130Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8298333Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8298831Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8299037Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8299245Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8299251Z 
2025-07-17T11:07:26.8299345Z SIGABRT(6), PID: 1822841, Thread 1823324: 
2025-07-17T11:07:26.8299812Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8300021Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8300224Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8300505Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8301003Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8301211Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8301501Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8301505Z 
2025-07-17T11:07:26.8301597Z SIGABRT(6), PID: 1822843, Thread 1823249: 
2025-07-17T11:07:26.8302065Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8302371Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8302587Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8302789Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8303282Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8303490Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8303703Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8303708Z 
2025-07-17T11:07:26.8303802Z SIGABRT(6), PID: 1822841, Thread 1823325: 
2025-07-17T11:07:26.8304269Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8304478Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8304680Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8304879Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8305373Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8305649Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8305860Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8305865Z 
2025-07-17T11:07:26.8305957Z SIGABRT(6), PID: 1822843, Thread 1823251: 
2025-07-17T11:07:26.8306427Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8306636Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8306840Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8307038Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8307533Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8307743Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8307953Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8307957Z 
2025-07-17T11:07:26.8308045Z SIGABRT(6), PID: 1822841, Thread 1823326: 
2025-07-17T11:07:26.8308519Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8308795Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8309000Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8309196Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8309695Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8309971Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8310184Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8310189Z 
2025-07-17T11:07:26.8310338Z SIGABRT(6), PID: 1822843, Thread 1823253: 
2025-07-17T11:07:26.8310809Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8311020Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8311224Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8311418Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8311916Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8312131Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8312340Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8312346Z 
2025-07-17T11:07:26.8312434Z SIGABRT(6), PID: 1822841, Thread 1823327: 
2025-07-17T11:07:26.8312906Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8313109Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8313312Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8313579Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8314080Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8314286Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8314500Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8314506Z 
2025-07-17T11:07:26.8314596Z SIGABRT(6), PID: 1822843, Thread 1823255: 
2025-07-17T11:07:26.8315070Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8315276Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8315480Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8315676Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8316172Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8316381Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8316592Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8316665Z 
2025-07-17T11:07:26.8316756Z SIGABRT(6), PID: 1822841, Thread 1823328: 
2025-07-17T11:07:26.8317228Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8317439Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8317708Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8317906Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8318397Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8318668Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8318885Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8318890Z 
2025-07-17T11:07:26.8318981Z SIGABRT(6), PID: 1822837, Thread 1827698: 
2025-07-17T11:07:26.8319452Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8319663Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8319866Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8320062Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8320437Z frame #4: <unknown function> + 0x5a02023 (0x7f0bd3518023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8321008Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f0bd351c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8321604Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f0bd35131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8322060Z frame #7: <unknown function> + 0x79d655 (0x7f0be4a03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8322305Z frame #8: <unknown function> + 0xdc253 (0x7f0b9ba4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8322521Z frame #9: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8322738Z frame #10: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8322742Z 
2025-07-17T11:07:26.8322834Z SIGABRT(6), PID: 1822840, Thread 1827718: 
2025-07-17T11:07:26.8323304Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8323516Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8323721Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8323927Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8324296Z frame #4: <unknown function> + 0x5a02023 (0x7fa30f22a023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8324858Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30f22e1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8325451Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30f2251d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8325905Z frame #7: <unknown function> + 0x79d655 (0x7fa320715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8326136Z frame #8: <unknown function> + 0xdc253 (0x7fa2d7760253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8326411Z frame #9: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8326626Z frame #10: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8326633Z 
2025-07-17T11:07:26.8326722Z SIGABRT(6), PID: 1822843, Thread 1823257: 
2025-07-17T11:07:26.8327260Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8327473Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8327677Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8327876Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8328384Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8328593Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8328805Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8328809Z 
2025-07-17T11:07:26.8328897Z SIGABRT(6), PID: 1822841, Thread 1823329: 
2025-07-17T11:07:26.8329376Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8329584Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8329788Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8329982Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8330547Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8330756Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8330967Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8330971Z 
2025-07-17T11:07:26.8331061Z SIGABRT(6), PID: 1822843, Thread 1823259: 
2025-07-17T11:07:26.8331532Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8331739Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8331944Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8332141Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8332653Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8332863Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8333077Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8333083Z 
2025-07-17T11:07:26.8333250Z SIGABRT(6), PID: 1822841, Thread 1823330: 
2025-07-17T11:07:26.8333720Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8333926Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8334130Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8334411Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8334910Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8335117Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8335388Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8335395Z 
2025-07-17T11:07:26.8335489Z SIGABRT(6), PID: 1822843, Thread 1823261: 
2025-07-17T11:07:26.8336048Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8336260Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8336470Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8336662Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8337158Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8337366Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8337577Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8337582Z 
2025-07-17T11:07:26.8337670Z SIGABRT(6), PID: 1822841, Thread 1823331: 
2025-07-17T11:07:26.8338140Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8338348Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8338750Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8338952Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8339451Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8339663Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8339870Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8339874Z 
2025-07-17T11:07:26.8339967Z SIGABRT(6), PID: 1822843, Thread 1823263: 
2025-07-17T11:07:26.8340434Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8340645Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8340848Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8341045Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8341538Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8341829Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8342039Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8342043Z 
2025-07-17T11:07:26.8342134Z SIGABRT(6), PID: 1822841, Thread 1823332: 
2025-07-17T11:07:26.8342612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8342895Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8343096Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8343292Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8343856Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8344070Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8344277Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8344281Z 
2025-07-17T11:07:26.8344373Z SIGABRT(6), PID: 1822843, Thread 1823265: 
2025-07-17T11:07:26.8344846Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8345055Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8345258Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8345468Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8345961Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8346179Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8346389Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8346395Z 
2025-07-17T11:07:26.8346490Z SIGABRT(6), PID: 1822841, Thread 1823333: 
2025-07-17T11:07:26.8347025Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8347239Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8347442Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8347649Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8348142Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8348352Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8348558Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8348566Z 
2025-07-17T11:07:26.8348663Z SIGABRT(6), PID: 1822843, Thread 1823267: 
2025-07-17T11:07:26.8349142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8349347Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8349559Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8349820Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8350317Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8350525Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8350803Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8350808Z 
2025-07-17T11:07:26.8350897Z SIGABRT(6), PID: 1822837, Thread 1827699: 
2025-07-17T11:07:26.8351385Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8351658Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8351866Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8352066Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8352448Z frame #4: <unknown function> + 0x5a02023 (0x7f0bd3518023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8353011Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f0bd351c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8353609Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f0bd35131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8353994Z frame #7: <unknown function> + 0x79d655 (0x7f0be4a03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8354232Z frame #8: <unknown function> + 0xdc253 (0x7f0b9ba4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8354440Z frame #9: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8354654Z frame #10: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8354744Z SIGABRT(6), PID: 1822841, Thread 1823334: 
2025-07-17T11:07:26.8355285Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8355496Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8355703Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8355926Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8356426Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8356634Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8356872Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8356878Z 
2025-07-17T11:07:26.8356882Z 
2025-07-17T11:07:26.8357002Z SIGABRT(6), PID: 1822840, Thread 1827719: 
2025-07-17T11:07:26.8357478Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8357714Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8357952Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8358215Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8358677Z frame #4: <unknown function> + 0x5a02023 (0x7fa30f22a023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8359255Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30f22e1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8359921Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30f2251d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8360313Z frame #7: <unknown function> + 0x79d655 (0x7fa320715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8360611Z frame #8: <unknown function> + 0xdc253 (0x7fa2d7760253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8360841Z frame #9: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8361061Z frame #10: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8361065Z 
2025-07-17T11:07:26.8361157Z SIGABRT(6), PID: 1822843, Thread 1823269: 
2025-07-17T11:07:26.8361637Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8361871Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8362074Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8362274Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8362792Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8362999Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8363212Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8363217Z 
2025-07-17T11:07:26.8363307Z SIGABRT(6), PID: 1822841, Thread 1823335: 
2025-07-17T11:07:26.8363848Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8364064Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8364270Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8364468Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8364985Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8365206Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8365415Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8365418Z 
2025-07-17T11:07:26.8365525Z SIGABRT(6), PID: 1822843, Thread 1823271: 
2025-07-17T11:07:26.8366006Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8366224Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8366434Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8366640Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8367248Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8367464Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8367674Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8367741Z 
2025-07-17T11:07:26.8367837Z SIGABRT(6), PID: 1822841, Thread 1823336: 
2025-07-17T11:07:26.8368307Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8368513Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8368775Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8368983Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8369480Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8369691Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8369905Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8369909Z 
2025-07-17T11:07:26.8370001Z SIGABRT(6), PID: 1822843, Thread 1823273: 
2025-07-17T11:07:26.8370468Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8370681Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8370881Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8371081Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8371575Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8371783Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8372054Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8372058Z 
2025-07-17T11:07:26.8372153Z SIGABRT(6), PID: 1822841, Thread 1823337: 
2025-07-17T11:07:26.8372625Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8372834Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8373037Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8373237Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8373729Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8373943Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8374150Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8374155Z 
2025-07-17T11:07:26.8374247Z SIGABRT(6), PID: 1822843, Thread 1823275: 
2025-07-17T11:07:26.8374715Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8374922Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8375198Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8375397Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8375889Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8376271Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8376481Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8376488Z 
2025-07-17T11:07:26.8376577Z SIGABRT(6), PID: 1822841, Thread 1823338: 
2025-07-17T11:07:26.8377124Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8377358Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8377564Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8377760Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8378261Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8378469Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8378681Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8378686Z 
2025-07-17T11:07:26.8378775Z SIGABRT(6), PID: 1822843, Thread 1823277: 
2025-07-17T11:07:26.8379246Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8379456Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8379660Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8379856Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8380432Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8380643Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8380853Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8380859Z 
2025-07-17T11:07:26.8380949Z SIGABRT(6), PID: 1822841, Thread 1823339: 
2025-07-17T11:07:26.8381430Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8381635Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8381838Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8382037Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8382531Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8382738Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8382950Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8383032Z 
2025-07-17T11:07:26.8383123Z SIGABRT(6), PID: 1822843, Thread 1823279: 
2025-07-17T11:07:26.8383596Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8383803Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8384010Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8384272Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8384771Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8385035Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8385252Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8385258Z 
2025-07-17T11:07:26.8385348Z SIGABRT(6), PID: 1822841, Thread 1823340: 
2025-07-17T11:07:26.8385817Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8386019Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8386228Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8386422Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8386918Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8387125Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8387336Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8387339Z 
2025-07-17T11:07:26.8387431Z SIGABRT(6), PID: 1822837, Thread 1827700: 
2025-07-17T11:07:26.8387908Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8388186Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8388395Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8388596Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8388977Z frame #4: <unknown function> + 0x5a02023 (0x7f0bd3518023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8389551Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f0bd351c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8390144Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f0bd35131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8390527Z frame #7: <unknown function> + 0x79d655 (0x7f0be4a03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8390759Z frame #8: <unknown function> + 0xdc253 (0x7f0b9ba4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8390971Z frame #9: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8391189Z frame #10: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8391194Z 
2025-07-17T11:07:26.8391284Z SIGABRT(6), PID: 1822843, Thread 1823281: 
2025-07-17T11:07:26.8391824Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8392036Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8392239Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8392507Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8393007Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8393217Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8393484Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8393582Z SIGABRT(6), PID: 1822840, Thread 1827720: 
2025-07-17T11:07:26.8394051Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8394261Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8394465Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8394671Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8395060Z frame #4: <unknown function> + 0x5a02023 (0x7fa30f22a023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8395627Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30f22e1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8396218Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30f2251d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8396599Z frame #7: <unknown function> + 0x79d655 (0x7fa320715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8396911Z frame #8: <unknown function> + 0xdc253 (0x7fa2d7760253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8397126Z frame #9: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8397344Z frame #10: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8397350Z 
2025-07-17T11:07:26.8397353Z 
2025-07-17T11:07:26.8397444Z SIGABRT(6), PID: 1822841, Thread 1823341: 
2025-07-17T11:07:26.8397916Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8398123Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8398327Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8398528Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8399033Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8399241Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8399449Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8399453Z 
2025-07-17T11:07:26.8399545Z SIGABRT(6), PID: 1822843, Thread 1823282: 
2025-07-17T11:07:26.8400080Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8400287Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8400489Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8400747Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8401247Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8401452Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8401721Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8401725Z 
2025-07-17T11:07:26.8401820Z SIGABRT(6), PID: 1822841, Thread 1823342: 
2025-07-17T11:07:26.8402295Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8402499Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8402704Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8402901Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8403399Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8403605Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8403815Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8403821Z 
2025-07-17T11:07:26.8403909Z SIGABRT(6), PID: 1822843, Thread 1823284: 
2025-07-17T11:07:26.8404377Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8404585Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8404855Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8405053Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8405546Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8405751Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8405959Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8405964Z 
2025-07-17T11:07:26.8406053Z SIGABRT(6), PID: 1822841, Thread 1823343: 
2025-07-17T11:07:26.8406525Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8406734Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8406937Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8407136Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8407630Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8407836Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8408109Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8408113Z 
2025-07-17T11:07:26.8408206Z SIGABRT(6), PID: 1822843, Thread 1823285: 
2025-07-17T11:07:26.8408676Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8408951Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8409154Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8409350Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8409899Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8410112Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8410322Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8410326Z 
2025-07-17T11:07:26.8410417Z SIGABRT(6), PID: 1822838, Thread 1822838: 
2025-07-17T11:07:26.8410885Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8411096Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8411328Z frame #2: <unknown function> + 0x7744b (0x7ff598fce44b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8411560Z frame #3: <unknown function> + 0x76c8e (0x7ff598fcdc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8411786Z frame #4: <unknown function> + 0x6ad69 (0x7ff598fc1d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8412003Z frame #5: <unknown function> + 0x3f54db (0x7ff629c264db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8412211Z frame #6: <unknown function> + 0x3e11d8 (0x7ff629c121d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8412429Z frame #7: <unknown function> + 0xb2f86 (0x7ff6298e3f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8412634Z frame #8: <unknown function> + 0x8e501 (0x7ff6298bf501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8413089Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7ff6546c67ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.8413478Z frame #10: <unknown function> + 0xd307d9 (0x7ff672c197d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8413708Z frame #11: <unknown function> + 0x139902 (0x55a55c967902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8413961Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55a55c95ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8414192Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8414435Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55a55c95ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8414663Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8414906Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8415130Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8415363Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8415586Z frame #19: <unknown function> + 0x1474b2 (0x55a55c9754b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8415789Z frame #20: PyObject_Call + 0xbc (0x55a55c975e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8416152Z frame #21: <unknown function> + 0x1ff69d (0x55a55ca2d69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8416374Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55a55c963023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8416621Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55a55c95ecc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8416938Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8417191Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55a55c95a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8417420Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8417736Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55a55c95a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8417963Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8418210Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55a55c95ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8418466Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55a55c9624b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8418685Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55a55c973ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8418907Z frame #32: <unknown function> + 0x205949 (0x55a55ca33949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8419135Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55a55c963023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8419376Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55a55c95ecc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8419601Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8419836Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55a55c95a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8420062Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8420295Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55a55c95a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8420518Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8420829Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55a55c95a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8421055Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8421297Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55a55c95b3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8421520Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8421917Z frame #44: <unknown function> + 0x7ab3f7 (0x7ff6726943f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8422141Z frame #45: <unknown function> + 0x13bc88 (0x55a55c969c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8422350Z frame #46: PyObject_Call + 0x20f (0x55a55c975f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8422594Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55a55c95f98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8422824Z frame #48: <unknown function> + 0x147608 (0x55a55c975608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8423064Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a55c95cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8423288Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8423524Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55a55c95a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8423744Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8424062Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55a55c95b3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8424286Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8424523Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55a55c95a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8424812Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8425051Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a55c95cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8425271Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8425512Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a55c95cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8425811Z frame #60: <unknown function> + 0x147608 (0x55a55c975608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8426056Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a55c95cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8426272Z frame #62: <unknown function> + 0x147608 (0x55a55c975608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8426277Z 
2025-07-17T11:07:26.8426376Z SIGABRT(6), PID: 1822841, Thread 1823344: 
2025-07-17T11:07:26.8426857Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8427076Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8427283Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8427486Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8427994Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8428207Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8428415Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8428420Z 
2025-07-17T11:07:26.8428514Z SIGABRT(6), PID: 1822843, Thread 1823287: 
2025-07-17T11:07:26.8429054Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8429275Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8429480Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8429682Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8430183Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8430390Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8430598Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8430604Z 
2025-07-17T11:07:26.8430702Z SIGABRT(6), PID: 1822838, Thread 1822936: 
2025-07-17T11:07:26.8431172Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8431383Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8431586Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8431786Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8432357Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8432570Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8432775Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8432848Z 
2025-07-17T11:07:26.8432941Z SIGABRT(6), PID: 1822841, Thread 1823345: 
2025-07-17T11:07:26.8433417Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8433618Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8433883Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8434087Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8434582Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8434787Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8434999Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8435003Z 
2025-07-17T11:07:26.8435091Z SIGABRT(6), PID: 1822843, Thread 1823289: 
2025-07-17T11:07:26.8435559Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8435766Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8435972Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8436166Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8436660Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8436929Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8437144Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8437148Z 
2025-07-17T11:07:26.8437237Z SIGABRT(6), PID: 1822841, Thread 1823346: 
2025-07-17T11:07:26.8437711Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8437913Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8438121Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8438315Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8438807Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8439013Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8439223Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8439311Z SIGABRT(6), PID: 1822838, Thread 1822939: 
2025-07-17T11:07:26.8439781Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8440054Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8440259Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8440455Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8440950Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8441218Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8441429Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8441433Z 
2025-07-17T11:07:26.8441437Z 
2025-07-17T11:07:26.8441526Z SIGABRT(6), PID: 1822843, Thread 1823291: 
2025-07-17T11:07:26.8442059Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8442269Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8442472Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8442665Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8443168Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8443373Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8443582Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8443586Z 
2025-07-17T11:07:26.8443677Z SIGABRT(6), PID: 1822840, Thread 1827721: 
2025-07-17T11:07:26.8444146Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8444357Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8444560Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8444760Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8445202Z frame #4: <unknown function> + 0x5a02023 (0x7fa30f22a023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8445771Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30f22e1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8446364Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30f2251d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8446750Z frame #7: <unknown function> + 0x79d655 (0x7fa320715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8446981Z frame #8: <unknown function> + 0xdc253 (0x7fa2d7760253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8447196Z frame #9: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8447411Z frame #10: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8447416Z 
2025-07-17T11:07:26.8447506Z SIGABRT(6), PID: 1822837, Thread 1827701: 
2025-07-17T11:07:26.8447985Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f0b9d7e1b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8448263Z frame #1: <unknown function> + 0x42520 (0x7f0c8f09a520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8448465Z frame #2: <unknown function> + 0x91117 (0x7f0c8f0e9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8448666Z frame #3: pthread_cond_wait + 0x211 (0x7f0c8f0eba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8449035Z frame #4: <unknown function> + 0x5a02023 (0x7f0bd3518023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8449662Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f0bd351c1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8450310Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f0bd35131d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8450701Z frame #7: <unknown function> + 0x79d655 (0x7f0be4a03655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8450937Z frame #8: <unknown function> + 0xdc253 (0x7f0b9ba4e253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8451149Z frame #9: <unknown function> + 0x94ac3 (0x7f0c8f0ecac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8451361Z frame #10: <unknown function> + 0x126850 (0x7f0c8f17e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8451455Z SIGABRT(6), PID: 1822841, Thread 1823347: 
2025-07-17T11:07:26.8451926Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8452135Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8452337Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8452537Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8453037Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8453244Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8453518Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8453526Z 
2025-07-17T11:07:26.8453530Z 
2025-07-17T11:07:26.8453621Z SIGABRT(6), PID: 1822843, Thread 1823293: 
2025-07-17T11:07:26.8454095Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8454303Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8454511Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8454706Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8455203Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8455410Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8455623Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8455627Z 
2025-07-17T11:07:26.8455715Z SIGABRT(6), PID: 1822838, Thread 1822943: 
2025-07-17T11:07:26.8456265Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8456476Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8456769Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8456970Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8463691Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8464140Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8464386Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8464392Z 
2025-07-17T11:07:26.8464497Z SIGABRT(6), PID: 1822841, Thread 1823348: 
2025-07-17T11:07:26.8465103Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8465335Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8465542Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8465750Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8466278Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8466491Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8466704Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8466708Z 
2025-07-17T11:07:26.8466804Z SIGABRT(6), PID: 1822843, Thread 1823294: 
2025-07-17T11:07:26.8467301Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8467513Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8467716Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8467920Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8468510Z frame #4: <unknown function> + 0x3475ab (0x7f1d02ec15ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8468727Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8468950Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8468954Z 
2025-07-17T11:07:26.8469047Z SIGABRT(6), PID: 1822838, Thread 1822946: 
2025-07-17T11:07:26.8469516Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8469723Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8469927Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8470125Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8470623Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8470829Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8471044Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8471133Z SIGABRT(6), PID: 1822843, Thread 1823902: 
2025-07-17T11:07:26.8471672Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8472125Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f1c1471f08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8472332Z frame #2: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8472599Z frame #3: pthread_kill + 0x12c (0x7f1d0602b9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8472763Z frame #4: raise + 0x16 (0x7f1d05fd7476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8472923Z frame #5: abort + 0xd3 (0x7f1d05fbd7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8473333Z frame #6: <unknown function> + 0x972cc (0x7f1b822a82cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8473570Z frame #7: <unknown function> + 0x95f47 (0x7f1b822a6f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8473800Z frame #8: <unknown function> + 0x34b71 (0x7f1b82245b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8474009Z frame #9: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8474225Z frame #10: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8474232Z 
2025-07-17T11:07:26.8474235Z 
2025-07-17T11:07:26.8474332Z SIGABRT(6), PID: 1822841, Thread 1823349: 
2025-07-17T11:07:26.8474800Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8475010Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8475214Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8475414Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8475912Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8476117Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8476326Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8476394Z 
2025-07-17T11:07:26.8476492Z SIGABRT(6), PID: 1822838, Thread 1822949: 
2025-07-17T11:07:26.8476957Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8477177Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8477391Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8477604Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8478119Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8478335Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8478548Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8478640Z SIGABRT(6), PID: 1822843, Thread 1827014: 
2025-07-17T11:07:26.8479114Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8479325Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8479493Z frame #2: __poll + 0x4f (0x7f1d060adbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8479972Z frame #3: <unknown function> + 0x489f (0x7f1d05bb989f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.8480201Z frame #4: <unknown function> + 0x1483bc (0x56103287e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8480448Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8480737Z frame #6: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8480972Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8481191Z frame #8: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8481497Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x561032866a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8481729Z frame #10: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8481976Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x561032864c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8482196Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8482431Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8482654Z frame #14: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8482885Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8483100Z frame #16: <unknown function> + 0x147734 (0x56103287d734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8483319Z frame #17: <unknown function> + 0x21ca82 (0x561032952a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8483532Z frame #18: <unknown function> + 0x21c9e4 (0x5610329529e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8483746Z frame #19: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8483957Z frame #20: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8484049Z SIGABRT(6), PID: 1822841, Thread 1823350: 
2025-07-17T11:07:26.8484587Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8484811Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8485016Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8485221Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8485724Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8485935Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8486141Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8486145Z 
2025-07-17T11:07:26.8486150Z 
2025-07-17T11:07:26.8486154Z 
2025-07-17T11:07:26.8486247Z SIGABRT(6), PID: 1822841, Thread 1823351: 
2025-07-17T11:07:26.8486717Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8486924Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8487125Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8487323Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8487891Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8488101Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8488309Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8488375Z 
2025-07-17T11:07:26.8488469Z SIGABRT(6), PID: 1822838, Thread 1822952: 
2025-07-17T11:07:26.8488937Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8489144Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8489407Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8489611Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8490106Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8490317Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8490528Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8490532Z 
2025-07-17T11:07:26.8490619Z SIGABRT(6), PID: 1822841, Thread 1823352: 
2025-07-17T11:07:26.8491085Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8491291Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8491496Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8491690Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8492185Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8492455Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8492675Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8492679Z 
2025-07-17T11:07:26.8492768Z SIGABRT(6), PID: 1822838, Thread 1822954: 
2025-07-17T11:07:26.8493241Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8493444Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8493650Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8493846Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8494340Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8494550Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8494757Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8494761Z 
2025-07-17T11:07:26.8494847Z SIGABRT(6), PID: 1822840, Thread 1827722: 
2025-07-17T11:07:26.8495318Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8495614Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8495823Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8496113Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8496496Z frame #4: <unknown function> + 0x5a02023 (0x7fa30f22a023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8497152Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30f22e1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8497826Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30f2251d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8498218Z frame #7: <unknown function> + 0x79d655 (0x7fa320715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8498454Z frame #8: <unknown function> + 0xdc253 (0x7fa2d7760253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8498671Z frame #9: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8498893Z frame #10: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8498898Z 
2025-07-17T11:07:26.8498992Z SIGABRT(6), PID: 1822841, Thread 1823353: 
2025-07-17T11:07:26.8499465Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8499678Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8499880Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8500085Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8500586Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8500795Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8501082Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8501087Z 
2025-07-17T11:07:26.8501183Z SIGABRT(6), PID: 1822843, Thread 1827417: 
2025-07-17T11:07:26.8501655Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8501866Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8502071Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8502296Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f1d060292dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8502736Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f1c40467048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8503161Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f1c40469bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8503392Z frame #6: <unknown function> + 0xdc253 (0x7f1c1298b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8503601Z frame #7: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8503818Z frame #8: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8503822Z 
2025-07-17T11:07:26.8503995Z SIGABRT(6), PID: 1822838, Thread 1822957: 
2025-07-17T11:07:26.8504464Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8504673Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8504876Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8505143Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8505640Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8505907Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8506122Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8506128Z 
2025-07-17T11:07:26.8506220Z SIGABRT(6), PID: 1822841, Thread 1823354: 
2025-07-17T11:07:26.8506685Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8506890Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8507093Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8507291Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8507795Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8508004Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8508211Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8508218Z 
2025-07-17T11:07:26.8508305Z SIGABRT(6), PID: 1822838, Thread 1822959: 
2025-07-17T11:07:26.8508770Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8509039Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8509248Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8509445Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8509943Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8510148Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8510358Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8510362Z 
2025-07-17T11:07:26.8510449Z SIGABRT(6), PID: 1822841, Thread 1823355: 
2025-07-17T11:07:26.8510916Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8511122Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8511324Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8511518Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8512013Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8512287Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8512496Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8512501Z 
2025-07-17T11:07:26.8512589Z SIGABRT(6), PID: 1822843, Thread 1827418: 
2025-07-17T11:07:26.8513055Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8513324Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8513528Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8513748Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f1d060292dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8514281Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f1c40461741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8514520Z frame #5: <unknown function> + 0xdc253 (0x7f1c1298b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8514726Z frame #6: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8514933Z frame #7: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8514939Z 
2025-07-17T11:07:26.8515032Z SIGABRT(6), PID: 1822843, Thread 1827419: 
2025-07-17T11:07:26.8515497Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8515704Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8515873Z frame #2: ioctl + 0x3f (0x7f1d060af94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8516115Z frame #3: <unknown function> + 0x13eef0 (0x7f1b8234fef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8516340Z frame #4: <unknown function> + 0x1378c4 (0x7f1b823488c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8516569Z frame #5: <unknown function> + 0x95b2b (0x7f1b822a6b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8516791Z frame #6: <unknown function> + 0x34b71 (0x7f1b82245b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8517064Z frame #7: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8517276Z frame #8: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8517368Z SIGABRT(6), PID: 1822838, Thread 1822963: 
2025-07-17T11:07:26.8517830Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8518042Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8518246Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8518448Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8518944Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8519159Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8519367Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8519458Z SIGABRT(6), PID: 1822841, Thread 1823357: 
2025-07-17T11:07:26.8519921Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8520202Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8520404Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8520601Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8521095Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8521364Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8521571Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8521577Z 
2025-07-17T11:07:26.8521581Z 
2025-07-17T11:07:26.8521585Z 
2025-07-17T11:07:26.8521729Z SIGABRT(6), PID: 1822840, Thread 1827723: 
2025-07-17T11:07:26.8522195Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8522407Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8522619Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8522815Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8523197Z frame #4: <unknown function> + 0x5a02023 (0x7fa30f22a023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8523766Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30f22e1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8524365Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30f2251d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8524743Z frame #7: <unknown function> + 0x79d655 (0x7fa320715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8524974Z frame #8: <unknown function> + 0xdc253 (0x7fa2d7760253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8525272Z frame #9: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8525493Z frame #10: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8525497Z 
2025-07-17T11:07:26.8525585Z SIGABRT(6), PID: 1822841, Thread 1823358: 
2025-07-17T11:07:26.8526060Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8526269Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8526473Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8526668Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8527163Z frame #4: <unknown function> + 0x3475ab (0x7f80b12ab5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8527368Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8527579Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8527583Z 
2025-07-17T11:07:26.8527670Z SIGABRT(6), PID: 1822838, Thread 1822966: 
2025-07-17T11:07:26.8528135Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8528413Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8528616Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8528814Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8529310Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8529579Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8529791Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8529795Z 
2025-07-17T11:07:26.8529882Z SIGABRT(6), PID: 1822843, Thread 1827650: 
2025-07-17T11:07:26.8530414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8530633Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8530801Z frame #2: __poll + 0x4f (0x7f1d060adbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8531019Z frame #3: <unknown function> + 0xd4540 (0x7f1bca4bf540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8531228Z frame #4: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8531437Z frame #5: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8531444Z 
2025-07-17T11:07:26.8531531Z SIGABRT(6), PID: 1822841, Thread 1823903: 
2025-07-17T11:07:26.8531999Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8532446Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f7fc2b0908a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8532658Z frame #2: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8532848Z frame #3: pthread_kill + 0x12c (0x7f80b44159fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8533014Z frame #4: raise + 0x16 (0x7f80b43c1476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8533235Z frame #5: abort + 0xd3 (0x7f80b43a77f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8533477Z frame #6: <unknown function> + 0x972cc (0x7f7f306922cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8533703Z frame #7: <unknown function> + 0x95f47 (0x7f7f30690f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8533929Z frame #8: <unknown function> + 0x34b71 (0x7f7f3062fb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8534137Z frame #9: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8534352Z frame #10: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8534356Z 
2025-07-17T11:07:26.8534445Z SIGABRT(6), PID: 1822843, Thread 1827658: 
2025-07-17T11:07:26.8534912Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8535122Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8535286Z frame #2: __poll + 0x4f (0x7f1d060adbcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8535500Z frame #3: <unknown function> + 0xd65a2 (0x7f1bca4c15a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8535707Z frame #4: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8535990Z frame #5: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8536172Z SIGABRT(6), PID: 1822838, Thread 1822969: 
2025-07-17T11:07:26.8536636Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8536848Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8537133Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8537340Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8537844Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8538126Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8538337Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8538344Z 
2025-07-17T11:07:26.8538347Z 
2025-07-17T11:07:26.8538437Z SIGABRT(6), PID: 1822843, Thread 1827668: 
2025-07-17T11:07:26.8538898Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8539104Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8539308Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8539506Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8539718Z frame #4: <unknown function> + 0xd1e7f (0x7f1bca4bce7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8539926Z frame #5: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8540133Z frame #6: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8540137Z 
2025-07-17T11:07:26.8540228Z SIGABRT(6), PID: 1822841, Thread 1827425: 
2025-07-17T11:07:26.8540687Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8540893Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8541132Z frame #2: __poll + 0x4f (0x7f80b4497bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8541541Z frame #3: <unknown function> + 0x489f (0x7f80b3fa389f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.8541769Z frame #4: <unknown function> + 0x1483bc (0x555c0fcb83bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8542018Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8542244Z frame #6: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8542476Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8542699Z frame #8: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8542951Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x555c0fca0a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8543184Z frame #10: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8543433Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x555c0fc9ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8543658Z frame #12: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8543896Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8544189Z frame #14: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8544423Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8544643Z frame #16: <unknown function> + 0x147734 (0x555c0fcb7734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8544862Z frame #17: <unknown function> + 0x21ca82 (0x555c0fd8ca82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8545142Z frame #18: <unknown function> + 0x21c9e4 (0x555c0fd8c9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8545354Z frame #19: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8545568Z frame #20: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8545572Z 
2025-07-17T11:07:26.8545732Z SIGABRT(6), PID: 1822838, Thread 1822972: 
2025-07-17T11:07:26.8546201Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8546418Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8546625Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8546829Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8547330Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8547540Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8547751Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8547757Z 
2025-07-17T11:07:26.8547846Z SIGABRT(6), PID: 1822843, Thread 1827677: 
2025-07-17T11:07:26.8548316Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8548522Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8548726Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8548994Z frame #3: <unknown function> + 0x9ca53 (0x7f1d06031a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8549251Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x56103284abc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8549473Z frame #5: <unknown function> + 0x1d314f (0x56103290914f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8549694Z frame #6: <unknown function> + 0x1d2f6b (0x561032908f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8549910Z frame #7: <unknown function> + 0x6df36 (0x5610327a3f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8550154Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8550374Z frame #9: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8550611Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8550837Z frame #11: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8551069Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8551288Z frame #13: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8551520Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8551740Z frame #15: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8552040Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8552257Z frame #17: <unknown function> + 0x147734 (0x56103287d734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8552476Z frame #18: <unknown function> + 0x21ca82 (0x561032952a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8552766Z frame #19: <unknown function> + 0x21c9e4 (0x5610329529e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8552975Z frame #20: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8553185Z frame #21: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8553189Z 
2025-07-17T11:07:26.8553283Z SIGABRT(6), PID: 1822838, Thread 1822976: 
2025-07-17T11:07:26.8553833Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8554054Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8554259Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8554460Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8554962Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8555172Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8555379Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8555384Z 
2025-07-17T11:07:26.8555477Z SIGABRT(6), PID: 1822843, Thread 1827691: 
2025-07-17T11:07:26.8555942Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8556151Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8556312Z frame #2: read + 0x4c (0x7f1d060a981c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8556492Z frame #3: _Py_read + 0x4c (0x561032910a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8556778Z frame #4: <unknown function> + 0x208c3b (0x56103293ec3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8557002Z frame #5: <unknown function> + 0x14648d (0x56103287c48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8557245Z frame #6: PyObject_VectorcallMethod + 0x85 (0x56103288b215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8557468Z frame #7: <unknown function> + 0x208ae7 (0x56103293eae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8557679Z frame #8: <unknown function> + 0x208472 (0x56103293e472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8557900Z frame #9: <unknown function> + 0x1da6ee (0x5610329106ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8558119Z frame #10: <unknown function> + 0x1483bc (0x56103287e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8558358Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8558582Z frame #12: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8558816Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x5610328623bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8559033Z frame #14: <unknown function> + 0x147734 (0x56103287d734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8559288Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x561032864c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8559509Z frame #16: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8559812Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8560030Z frame #18: _PyFunction_Vectorcall + 0x6c (0x5610328720ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8560261Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x5610328627b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8560546Z frame #20: <unknown function> + 0x147734 (0x56103287d734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8560768Z frame #21: <unknown function> + 0x21ca82 (0x561032952a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8560981Z frame #22: <unknown function> + 0x21c9e4 (0x5610329529e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8561250Z frame #23: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8561466Z frame #24: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8561560Z SIGABRT(6), PID: 1822838, Thread 1822979: 
2025-07-17T11:07:26.8562025Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8562236Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8562445Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8562646Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8563139Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8563347Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8563553Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8563559Z 
2025-07-17T11:07:26.8563562Z 
2025-07-17T11:07:26.8563653Z SIGABRT(6), PID: 1822841, Thread 1827426: 
2025-07-17T11:07:26.8564114Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8564321Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8564589Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8564815Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f80b44132dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8565257Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f7fee851048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8565679Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f7fee853bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8565910Z frame #6: <unknown function> + 0xdc253 (0x7f7fc0d75253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8566118Z frame #7: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8566322Z frame #8: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8566328Z 
2025-07-17T11:07:26.8566419Z SIGABRT(6), PID: 1822840, Thread 1827724: 
2025-07-17T11:07:26.8566887Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8567094Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8567300Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8567569Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8567942Z frame #4: <unknown function> + 0x5a02023 (0x7fa30f22a023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8568509Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30f22e1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8569181Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30f2251d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8569626Z frame #7: <unknown function> + 0x79d655 (0x7fa320715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8569863Z frame #8: <unknown function> + 0xdc253 (0x7fa2d7760253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8570080Z frame #9: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8570295Z frame #10: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8570298Z 
2025-07-17T11:07:26.8570387Z SIGABRT(6), PID: 1822838, Thread 1822981: 
2025-07-17T11:07:26.8570867Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8571078Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8571282Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8571488Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8571992Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8572200Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8572409Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8572414Z 
2025-07-17T11:07:26.8572502Z SIGABRT(6), PID: 1822838, Thread 1822984: 
2025-07-17T11:07:26.8573037Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8573247Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8573452Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8573652Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8574153Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8574362Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8574571Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8574577Z 
2025-07-17T11:07:26.8574667Z SIGABRT(6), PID: 1822841, Thread 1827427: 
2025-07-17T11:07:26.8575138Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8575341Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8575546Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8575766Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f80b44132dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8576378Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f7fee84b741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8576607Z frame #5: <unknown function> + 0xdc253 (0x7f7fc0d75253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8576816Z frame #6: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8577111Z frame #7: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8577116Z 
2025-07-17T11:07:26.8577207Z SIGABRT(6), PID: 1822841, Thread 1827430: 
2025-07-17T11:07:26.8577674Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8577956Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8578128Z frame #2: ioctl + 0x3f (0x7f80b449994f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8578366Z frame #3: <unknown function> + 0x13eef0 (0x7f7f30739ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8578594Z frame #4: <unknown function> + 0x1378c4 (0x7f7f307328c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8578823Z frame #5: <unknown function> + 0x95b2b (0x7f7f30690b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8579048Z frame #6: <unknown function> + 0x34b71 (0x7f7f3062fb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8579258Z frame #7: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8579465Z frame #8: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8579558Z SIGABRT(6), PID: 1822838, Thread 1822987: 
2025-07-17T11:07:26.8580022Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8580231Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8580433Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8580633Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8581213Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8581424Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8581631Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8581642Z 
2025-07-17T11:07:26.8581645Z 
2025-07-17T11:07:26.8581733Z SIGABRT(6), PID: 1822838, Thread 1822991: 
2025-07-17T11:07:26.8582200Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8582405Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8582607Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8582806Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8583308Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8583513Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8583724Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8583808Z 
2025-07-17T11:07:26.8583897Z SIGABRT(6), PID: 1822841, Thread 1827646: 
2025-07-17T11:07:26.8584364Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8584567Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8584820Z frame #2: __poll + 0x4f (0x7f80b4497bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8585033Z frame #3: <unknown function> + 0xd4540 (0x7f7f788a9540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8585236Z frame #4: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8585442Z frame #5: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8585446Z 
2025-07-17T11:07:26.8585596Z SIGABRT(6), PID: 1822838, Thread 1822995: 
2025-07-17T11:07:26.8586060Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8586272Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8586474Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8586674Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8587169Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8587379Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8587586Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8587591Z 
2025-07-17T11:07:26.8587682Z SIGABRT(6), PID: 1822843, Thread 1827742: 
2025-07-17T11:07:26.8588142Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8588352Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8588551Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8588816Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8589195Z frame #4: <unknown function> + 0x5a02023 (0x7f1c4a455023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8589772Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1c4a4591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8590366Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1c4a4501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8590752Z frame #7: <unknown function> + 0x79d655 (0x7f1c5b940655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8590982Z frame #8: <unknown function> + 0xdc253 (0x7f1c1298b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8591194Z frame #9: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8591408Z frame #10: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8591415Z 
2025-07-17T11:07:26.8591504Z SIGABRT(6), PID: 1822841, Thread 1827654: 
2025-07-17T11:07:26.8591980Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8592254Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8592420Z frame #2: __poll + 0x4f (0x7f80b4497bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8592633Z frame #3: <unknown function> + 0xd65a2 (0x7f7f788ab5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8592848Z frame #4: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8593115Z frame #5: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8593119Z 
2025-07-17T11:07:26.8593209Z SIGABRT(6), PID: 1822840, Thread 1827725: 
2025-07-17T11:07:26.8593673Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fa2d94f3b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8593944Z frame #1: <unknown function> + 0x42520 (0x7fa3cadaa520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8594153Z frame #2: <unknown function> + 0x91117 (0x7fa3cadf9117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8594359Z frame #3: pthread_cond_wait + 0x211 (0x7fa3cadfba41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8594734Z frame #4: <unknown function> + 0x5a02023 (0x7fa30f22a023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8595296Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fa30f22e1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8595885Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fa30f2251d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8596266Z frame #7: <unknown function> + 0x79d655 (0x7fa320715655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8596495Z frame #8: <unknown function> + 0xdc253 (0x7fa2d7760253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8596705Z frame #9: <unknown function> + 0x94ac3 (0x7fa3cadfcac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8596918Z frame #10: <unknown function> + 0x126850 (0x7fa3cae8e850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8596924Z 
2025-07-17T11:07:26.8597015Z SIGABRT(6), PID: 1822838, Thread 1822998: 
2025-07-17T11:07:26.8597549Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8597762Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8597965Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8598165Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8598670Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8598878Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8599087Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8599093Z 
2025-07-17T11:07:26.8599188Z SIGABRT(6), PID: 1822841, Thread 1827664: 
2025-07-17T11:07:26.8599649Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8599860Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8600062Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8600332Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8600546Z frame #4: <unknown function> + 0xd1e7f (0x7f7f788a6e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8600752Z frame #5: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8600956Z frame #6: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8601124Z 
2025-07-17T11:07:26.8601219Z SIGABRT(6), PID: 1822838, Thread 1823002: 
2025-07-17T11:07:26.8601684Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8601894Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8602157Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8602358Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8602863Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8603070Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8603284Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8603288Z 
2025-07-17T11:07:26.8603376Z SIGABRT(6), PID: 1822841, Thread 1827671: 
2025-07-17T11:07:26.8603844Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8604047Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8604250Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8604453Z frame #3: <unknown function> + 0x9ca53 (0x7f80b441ba53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8604706Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x555c0fc84bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8604930Z frame #5: <unknown function> + 0x1d314f (0x555c0fd4314f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8605219Z frame #6: <unknown function> + 0x1d2f6b (0x555c0fd42f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8605440Z frame #7: <unknown function> + 0x6df36 (0x555c0fbddf36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8605687Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8605912Z frame #9: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8606154Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8606384Z frame #11: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8606618Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8606839Z frame #13: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8607076Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8607297Z frame #15: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8607529Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8607746Z frame #17: <unknown function> + 0x147734 (0x555c0fcb7734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8607969Z frame #18: <unknown function> + 0x21ca82 (0x555c0fd8ca82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8608255Z frame #19: <unknown function> + 0x21c9e4 (0x555c0fd8c9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8608465Z frame #20: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8608671Z frame #21: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8608733Z 
2025-07-17T11:07:26.8608829Z SIGABRT(6), PID: 1822838, Thread 1823006: 
2025-07-17T11:07:26.8609296Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8609506Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8609770Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8609978Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8610478Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8610686Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8610892Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8610898Z 
2025-07-17T11:07:26.8610991Z SIGABRT(6), PID: 1822841, Thread 1827682: 
2025-07-17T11:07:26.8611459Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8611668Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8611832Z frame #2: read + 0x4c (0x7f80b449381c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8612016Z frame #3: _Py_read + 0x4c (0x555c0fd4aa6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8612238Z frame #4: <unknown function> + 0x208c3b (0x555c0fd78c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8612461Z frame #5: <unknown function> + 0x14648d (0x555c0fcb648d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8612703Z frame #6: PyObject_VectorcallMethod + 0x85 (0x555c0fcc5215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8612989Z frame #7: <unknown function> + 0x208ae7 (0x555c0fd78ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8613208Z frame #8: <unknown function> + 0x208472 (0x555c0fd78472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8613429Z frame #9: <unknown function> + 0x1da6ee (0x555c0fd4a6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8613652Z frame #10: <unknown function> + 0x1483bc (0x555c0fcb83bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8613891Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8614115Z frame #12: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8614350Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x555c0fc9c3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8614567Z frame #14: <unknown function> + 0x147734 (0x555c0fcb7734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8614819Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x555c0fc9ec55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8615042Z frame #16: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8615282Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8615504Z frame #18: _PyFunction_Vectorcall + 0x6c (0x555c0fcac0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8615736Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x555c0fc9c7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8616224Z frame #20: <unknown function> + 0x147734 (0x555c0fcb7734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8616450Z frame #21: <unknown function> + 0x21ca82 (0x555c0fd8ca82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8616667Z frame #22: <unknown function> + 0x21c9e4 (0x555c0fd8c9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8616984Z frame #23: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8617198Z frame #24: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8617203Z 
2025-07-17T11:07:26.8617295Z SIGABRT(6), PID: 1822838, Thread 1823010: 
2025-07-17T11:07:26.8617836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8618051Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8618258Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8618461Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8618959Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8619171Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8619380Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8619384Z 
2025-07-17T11:07:26.8619475Z SIGABRT(6), PID: 1822838, Thread 1823014: 
2025-07-17T11:07:26.8619940Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8620149Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8620348Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8620548Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8621123Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8621339Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8621549Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8621553Z 
2025-07-17T11:07:26.8621645Z SIGABRT(6), PID: 1822838, Thread 1823018: 
2025-07-17T11:07:26.8622112Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8622324Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8622525Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8622724Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8623229Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8623436Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8623640Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8623647Z 
2025-07-17T11:07:26.8623736Z SIGABRT(6), PID: 1822838, Thread 1823022: 
2025-07-17T11:07:26.8624281Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8624485Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8624689Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8624951Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8625447Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8625653Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8625923Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8625928Z 
2025-07-17T11:07:26.8626020Z SIGABRT(6), PID: 1822843, Thread 1827743: 
2025-07-17T11:07:26.8626486Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8626694Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8626894Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8627091Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8627465Z frame #4: <unknown function> + 0x5a02023 (0x7f1c4a455023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8628031Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1c4a4591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8628629Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1c4a4501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8629008Z frame #7: <unknown function> + 0x79d655 (0x7f1c5b940655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8629239Z frame #8: <unknown function> + 0xdc253 (0x7f1c1298b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8629513Z frame #9: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8629734Z frame #10: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8629738Z 
2025-07-17T11:07:26.8629828Z SIGABRT(6), PID: 1822838, Thread 1823026: 
2025-07-17T11:07:26.8630304Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8630509Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8630712Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8630910Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8631422Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8631629Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8631837Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8631841Z 
2025-07-17T11:07:26.8631930Z SIGABRT(6), PID: 1822838, Thread 1823029: 
2025-07-17T11:07:26.8632402Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8632671Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8632885Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8633081Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8633650Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8633866Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8634074Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8634151Z 
2025-07-17T11:07:26.8634249Z SIGABRT(6), PID: 1822841, Thread 1827702: 
2025-07-17T11:07:26.8634726Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8634936Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8635137Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8635340Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8635711Z frame #4: <unknown function> + 0x5a02023 (0x7f7ff883f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8636290Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7ff88431ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8636883Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7ff883a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8637264Z frame #7: <unknown function> + 0x79d655 (0x7f8009d2a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8637496Z frame #8: <unknown function> + 0xdc253 (0x7f7fc0d75253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8637780Z frame #9: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8638000Z frame #10: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8638005Z 
2025-07-17T11:07:26.8638096Z SIGABRT(6), PID: 1822838, Thread 1823033: 
2025-07-17T11:07:26.8638564Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8638773Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8638974Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8639171Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8639666Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8639876Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8640096Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8640100Z 
2025-07-17T11:07:26.8640192Z SIGABRT(6), PID: 1822838, Thread 1823037: 
2025-07-17T11:07:26.8640655Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8640929Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8641148Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8641361Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8641925Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8642137Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8642359Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8642363Z 
2025-07-17T11:07:26.8642518Z SIGABRT(6), PID: 1822838, Thread 1823041: 
2025-07-17T11:07:26.8643001Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8643218Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8643422Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8643629Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8644132Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8644353Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8644574Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8644578Z 
2025-07-17T11:07:26.8644674Z SIGABRT(6), PID: 1822843, Thread 1827744: 
2025-07-17T11:07:26.8645141Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8645346Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8645549Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8645832Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8646203Z frame #4: <unknown function> + 0x5a02023 (0x7f1c4a455023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8646760Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1c4a4591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8647350Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1c4a4501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8647729Z frame #7: <unknown function> + 0x79d655 (0x7f1c5b940655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8647961Z frame #8: <unknown function> + 0xdc253 (0x7f1c1298b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8648169Z frame #9: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8648385Z frame #10: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8648390Z 
2025-07-17T11:07:26.8648478Z SIGABRT(6), PID: 1822838, Thread 1823045: 
2025-07-17T11:07:26.8648948Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8649219Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8649424Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8649623Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8650122Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8650392Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8650601Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8650606Z 
2025-07-17T11:07:26.8650692Z SIGABRT(6), PID: 1822838, Thread 1823049: 
2025-07-17T11:07:26.8651212Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8651420Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8651626Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8651822Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8652322Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8652529Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8652741Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8652745Z 
2025-07-17T11:07:26.8652835Z SIGABRT(6), PID: 1822838, Thread 1823053: 
2025-07-17T11:07:26.8653302Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8653509Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8653709Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8653908Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8654460Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8654671Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8654885Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8654889Z 
2025-07-17T11:07:26.8654981Z SIGABRT(6), PID: 1822841, Thread 1827703: 
2025-07-17T11:07:26.8655446Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8655651Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8655852Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8656132Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8656503Z frame #4: <unknown function> + 0x5a02023 (0x7f7ff883f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8657062Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7ff88431ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8657741Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7ff883a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8658123Z frame #7: <unknown function> + 0x79d655 (0x7f8009d2a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8658428Z frame #8: <unknown function> + 0xdc253 (0x7f7fc0d75253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8658638Z frame #9: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8658851Z frame #10: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8658855Z 
2025-07-17T11:07:26.8658949Z SIGABRT(6), PID: 1822838, Thread 1823057: 
2025-07-17T11:07:26.8659497Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8659718Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8659922Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8660125Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8660627Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8660838Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8661049Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8661053Z 
2025-07-17T11:07:26.8661145Z SIGABRT(6), PID: 1822838, Thread 1823061: 
2025-07-17T11:07:26.8661606Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8661813Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8662013Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8662210Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8662783Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8662995Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8663209Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8663217Z 
2025-07-17T11:07:26.8663307Z SIGABRT(6), PID: 1822838, Thread 1823064: 
2025-07-17T11:07:26.8663774Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8663978Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8664189Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8664388Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8664887Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8665091Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8665303Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8665371Z 
2025-07-17T11:07:26.8665468Z SIGABRT(6), PID: 1822843, Thread 1827745: 
2025-07-17T11:07:26.8665939Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8666147Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8666351Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8666608Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8666983Z frame #4: <unknown function> + 0x5a02023 (0x7f1c4a455023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8667606Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1c4a4591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8668204Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1c4a4501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8668584Z frame #7: <unknown function> + 0x79d655 (0x7f1c5b940655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8668822Z frame #8: <unknown function> + 0xdc253 (0x7f1c1298b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8669041Z frame #9: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8669262Z frame #10: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8669266Z 
2025-07-17T11:07:26.8669358Z SIGABRT(6), PID: 1822838, Thread 1823067: 
2025-07-17T11:07:26.8669856Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8670068Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8670276Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8670474Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8671052Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8671274Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8671492Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8671496Z 
2025-07-17T11:07:26.8671591Z SIGABRT(6), PID: 1822838, Thread 1823070: 
2025-07-17T11:07:26.8672064Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8672272Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8672478Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8672677Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8673190Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8673396Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8673609Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8673613Z 
2025-07-17T11:07:26.8673705Z SIGABRT(6), PID: 1822841, Thread 1827704: 
2025-07-17T11:07:26.8674243Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8674449Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8674652Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8674914Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8675289Z frame #4: <unknown function> + 0x5a02023 (0x7f7ff883f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8675936Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7ff88431ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8676534Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7ff883a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8676919Z frame #7: <unknown function> + 0x79d655 (0x7f8009d2a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8677149Z frame #8: <unknown function> + 0xdc253 (0x7f7fc0d75253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8677364Z frame #9: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8677577Z frame #10: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8677581Z 
2025-07-17T11:07:26.8677673Z SIGABRT(6), PID: 1822838, Thread 1823073: 
2025-07-17T11:07:26.8678143Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8678354Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8678556Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8678754Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8679317Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8679534Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8679743Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8679748Z 
2025-07-17T11:07:26.8679839Z SIGABRT(6), PID: 1822838, Thread 1823076: 
2025-07-17T11:07:26.8680301Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8680512Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8680714Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8680912Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8681407Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8681615Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8681823Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8681827Z 
2025-07-17T11:07:26.8681919Z SIGABRT(6), PID: 1822838, Thread 1823078: 
2025-07-17T11:07:26.8682446Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8682655Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8682855Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8683116Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8683610Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8683821Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8684089Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8684093Z 
2025-07-17T11:07:26.8684186Z SIGABRT(6), PID: 1822843, Thread 1827747: 
2025-07-17T11:07:26.8684652Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8684858Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8685061Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8685263Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8685634Z frame #4: <unknown function> + 0x5a02023 (0x7f1c4a455023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8686197Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1c4a4591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8686795Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1c4a4501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8687172Z frame #7: <unknown function> + 0x79d655 (0x7f1c5b940655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8687402Z frame #8: <unknown function> + 0xdc253 (0x7f1c1298b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8687672Z frame #9: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8687897Z frame #10: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8687902Z 
2025-07-17T11:07:26.8687990Z SIGABRT(6), PID: 1822838, Thread 1823081: 
2025-07-17T11:07:26.8688463Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8688671Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8688876Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8689074Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8689571Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8689779Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8689989Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8689994Z 
2025-07-17T11:07:26.8690085Z SIGABRT(6), PID: 1822838, Thread 1823084: 
2025-07-17T11:07:26.8690554Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8690831Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8691037Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8691232Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8691795Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8692000Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8692211Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8692277Z 
2025-07-17T11:07:26.8692371Z SIGABRT(6), PID: 1822841, Thread 1827705: 
2025-07-17T11:07:26.8692843Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8693050Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8693252Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8693452Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8693821Z frame #4: <unknown function> + 0x5a02023 (0x7f7ff883f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8694386Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7ff88431ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8694977Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7ff883a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8695361Z frame #7: <unknown function> + 0x79d655 (0x7f8009d2a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8695589Z frame #8: <unknown function> + 0xdc253 (0x7f7fc0d75253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8695864Z frame #9: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8696136Z frame #10: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8696141Z 
2025-07-17T11:07:26.8696233Z SIGABRT(6), PID: 1822838, Thread 1823087: 
2025-07-17T11:07:26.8696706Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8696919Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8697124Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8697324Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8697825Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8698038Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8698246Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8698250Z 
2025-07-17T11:07:26.8698342Z SIGABRT(6), PID: 1822843, Thread 1827749: 
2025-07-17T11:07:26.8698809Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8699118Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8699320Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8699519Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8699966Z frame #4: <unknown function> + 0x5a02023 (0x7f1c4a455023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8700530Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1c4a4591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8701190Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1c4a4501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8701576Z frame #7: <unknown function> + 0x79d655 (0x7f1c5b940655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8701804Z frame #8: <unknown function> + 0xdc253 (0x7f1c1298b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8702018Z frame #9: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8702234Z frame #10: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8702240Z 
2025-07-17T11:07:26.8702327Z SIGABRT(6), PID: 1822838, Thread 1823090: 
2025-07-17T11:07:26.8702795Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8703003Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8703217Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8703414Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8703914Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8704199Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8704413Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8704503Z SIGABRT(6), PID: 1822839, Thread 1822839: 
2025-07-17T11:07:26.8704969Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8705174Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8705411Z frame #2: <unknown function> + 0x7741f (0x7f6b7f2d241f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8705640Z frame #3: <unknown function> + 0x76c8e (0x7f6b7f2d1c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8705877Z frame #4: <unknown function> + 0x6ad69 (0x7f6b7f2c5d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8706098Z frame #5: <unknown function> + 0x3f54db (0x7f6c0ff2a4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8706316Z frame #6: <unknown function> + 0x3e11d8 (0x7f6c0ff161d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8706525Z frame #7: <unknown function> + 0xb2f86 (0x7f6c0fbe7f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8706734Z frame #8: <unknown function> + 0x8e501 (0x7f6c0fbc3501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8707124Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f6c3a9ca7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.8707583Z frame #10: <unknown function> + 0xd307d9 (0x7f6c58f1d7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8707812Z frame #11: <unknown function> + 0x139902 (0x55b30f51c902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8708066Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55b30f513a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8708380Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8708625Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55b30f513a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8708853Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8709157Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8709383Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8709625Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8709846Z frame #19: <unknown function> + 0x1474b2 (0x55b30f52a4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8710053Z frame #20: PyObject_Call + 0xbc (0x55b30f52ae4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8710276Z frame #21: <unknown function> + 0x1ff69d (0x55b30f5e269d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8710501Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55b30f518023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8710740Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55b30f513cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8710965Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8711200Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55b30f50f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8711424Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8711656Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55b30f50f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8711885Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8712188Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55b30f513a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8712451Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55b30f5174b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8712669Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55b30f528ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8712889Z frame #32: <unknown function> + 0x205949 (0x55b30f5e8949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8713120Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55b30f518023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8713361Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55b30f513cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8713581Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8713819Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55b30f50f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8714042Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8714277Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55b30f50f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8714500Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8714732Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55b30f50f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8714955Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8715259Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55b30f5103f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8715483Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8715871Z frame #44: <unknown function> + 0x7ab3f7 (0x7f6c589983f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8716151Z frame #45: <unknown function> + 0x13bc88 (0x55b30f51ec88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8716361Z frame #46: PyObject_Call + 0x20f (0x55b30f52af9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8716599Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55b30f51498d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8716819Z frame #48: <unknown function> + 0x147608 (0x55b30f52a608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8717122Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b30f511c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8717353Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8717589Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55b30f50f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8717814Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8718053Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55b30f5103f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8718279Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8718517Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55b30f50f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8718743Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8718979Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b30f511c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8719207Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8719441Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b30f511c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8719657Z frame #60: <unknown function> + 0x147608 (0x55b30f52a608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8719958Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b30f511c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8720183Z frame #62: <unknown function> + 0x147608 (0x55b30f52a608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8720186Z 
2025-07-17T11:07:26.8720190Z 
2025-07-17T11:07:26.8720282Z SIGABRT(6), PID: 1822838, Thread 1823093: 
2025-07-17T11:07:26.8720769Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8720981Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8721193Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8721393Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8721899Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8722110Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8722323Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8722327Z 
2025-07-17T11:07:26.8722417Z SIGABRT(6), PID: 1822839, Thread 1822961: 
2025-07-17T11:07:26.8722895Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8723176Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8723385Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8723580Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8724081Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8724350Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8724562Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8724566Z 
2025-07-17T11:07:26.8724714Z SIGABRT(6), PID: 1822841, Thread 1827706: 
2025-07-17T11:07:26.8725192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8725400Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8725605Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8725799Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8726183Z frame #4: <unknown function> + 0x5a02023 (0x7f7ff883f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8726747Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7ff88431ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8727348Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7ff883a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8727729Z frame #7: <unknown function> + 0x79d655 (0x7f8009d2a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8727958Z frame #8: <unknown function> + 0xdc253 (0x7f7fc0d75253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8728168Z frame #9: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8728445Z frame #10: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8728541Z SIGABRT(6), PID: 1822838, Thread 1823096: 
2025-07-17T11:07:26.8729014Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8729227Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8729434Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8729633Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8730135Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8730347Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8730553Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8730557Z 
2025-07-17T11:07:26.8730561Z 
2025-07-17T11:07:26.8730653Z SIGABRT(6), PID: 1822839, Thread 1822965: 
2025-07-17T11:07:26.8731122Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8731512Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8731713Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8731908Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8732408Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8732679Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8732888Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8732892Z 
2025-07-17T11:07:26.8732991Z SIGABRT(6), PID: 1822838, Thread 1823099: 
2025-07-17T11:07:26.8733520Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8733735Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8733939Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8734141Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8734635Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8734845Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8735049Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8735053Z 
2025-07-17T11:07:26.8735145Z SIGABRT(6), PID: 1822839, Thread 1822968: 
2025-07-17T11:07:26.8735612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8735821Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8736106Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8736303Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8736882Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8737094Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8737301Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8737305Z 
2025-07-17T11:07:26.8737397Z SIGABRT(6), PID: 1822838, Thread 1823102: 
2025-07-17T11:07:26.8737861Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8738068Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8738270Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8738471Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8738965Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8739173Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8739386Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8739489Z 
2025-07-17T11:07:26.8739580Z SIGABRT(6), PID: 1822839, Thread 1822971: 
2025-07-17T11:07:26.8740053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8740255Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8740538Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8740732Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8741229Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8741508Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8741720Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8741727Z 
2025-07-17T11:07:26.8741815Z SIGABRT(6), PID: 1822838, Thread 1823105: 
2025-07-17T11:07:26.8742282Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8742485Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8742693Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8742888Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8743386Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8743592Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8743802Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8743806Z 
2025-07-17T11:07:26.8743893Z SIGABRT(6), PID: 1822843, Thread 1827752: 
2025-07-17T11:07:26.8744359Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8744632Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8744843Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8745036Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8745419Z frame #4: <unknown function> + 0x5a02023 (0x7f1c4a455023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8745985Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1c4a4591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8746584Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1c4a4501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8746969Z frame #7: <unknown function> + 0x79d655 (0x7f1c5b940655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8747200Z frame #8: <unknown function> + 0xdc253 (0x7f1c1298b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8747406Z frame #9: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8747623Z frame #10: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8747628Z 
2025-07-17T11:07:26.8747781Z SIGABRT(6), PID: 1822839, Thread 1822974: 
2025-07-17T11:07:26.8748261Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8748469Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8748673Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8748930Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8749427Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8749634Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8749901Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8749907Z 
2025-07-17T11:07:26.8749999Z SIGABRT(6), PID: 1822838, Thread 1823108: 
2025-07-17T11:07:26.8750470Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8750677Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8750884Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8751083Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8751572Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8751783Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8751993Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8751997Z 
2025-07-17T11:07:26.8752087Z SIGABRT(6), PID: 1822839, Thread 1822977: 
2025-07-17T11:07:26.8752548Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8752752Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8753020Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8753219Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8753715Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8753921Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8754128Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8754132Z 
2025-07-17T11:07:26.8754221Z SIGABRT(6), PID: 1822839, Thread 1822980: 
2025-07-17T11:07:26.8754684Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8754892Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8755094Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8755290Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8755784Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8756063Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8756269Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8756274Z 
2025-07-17T11:07:26.8756366Z SIGABRT(6), PID: 1822838, Thread 1823112: 
2025-07-17T11:07:26.8756831Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8757105Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8757307Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8757507Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8758058Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8758273Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8758480Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8758571Z SIGABRT(6), PID: 1822841, Thread 1827708: 
2025-07-17T11:07:26.8759041Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8759247Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8759446Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8759648Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8760016Z frame #4: <unknown function> + 0x5a02023 (0x7f7ff883f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8760581Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7ff88431ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8761236Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7ff883a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8761619Z frame #7: <unknown function> + 0x79d655 (0x7f8009d2a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8761856Z frame #8: <unknown function> + 0xdc253 (0x7f7fc0d75253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8762062Z frame #9: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8762278Z frame #10: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8762282Z 
2025-07-17T11:07:26.8762285Z 
2025-07-17T11:07:26.8762372Z SIGABRT(6), PID: 1822839, Thread 1822983: 
2025-07-17T11:07:26.8762848Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8763057Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8763261Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8763456Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8763961Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8764166Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8764442Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8764446Z 
2025-07-17T11:07:26.8764533Z SIGABRT(6), PID: 1822838, Thread 1823116: 
2025-07-17T11:07:26.8765003Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8765273Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8765482Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8765677Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8766241Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8766453Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8766663Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8766667Z 
2025-07-17T11:07:26.8766755Z SIGABRT(6), PID: 1822839, Thread 1822986: 
2025-07-17T11:07:26.8767226Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8767434Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8767641Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8767835Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8768330Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8768536Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8768747Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8768751Z 
2025-07-17T11:07:26.8768839Z SIGABRT(6), PID: 1822838, Thread 1823120: 
2025-07-17T11:07:26.8769391Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8769603Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8769808Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8770005Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8770500Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8770713Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8770918Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8770924Z 
2025-07-17T11:07:26.8771015Z SIGABRT(6), PID: 1822839, Thread 1822988: 
2025-07-17T11:07:26.8771478Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8771684Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8771887Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8772082Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8772643Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8772848Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8773051Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8773125Z 
2025-07-17T11:07:26.8773220Z SIGABRT(6), PID: 1822839, Thread 1822992: 
2025-07-17T11:07:26.8773683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8773890Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8774151Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8774352Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8774847Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8775054Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8775261Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8775264Z 
2025-07-17T11:07:26.8775358Z SIGABRT(6), PID: 1822838, Thread 1823126: 
2025-07-17T11:07:26.8775829Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8776121Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8776329Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8776528Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8777018Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8777310Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8777523Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8777529Z 
2025-07-17T11:07:26.8777621Z SIGABRT(6), PID: 1822843, Thread 1827755: 
2025-07-17T11:07:26.8778092Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f1c1471eb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8778302Z frame #1: <unknown function> + 0x42520 (0x7f1d05fd7520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8778504Z frame #2: <unknown function> + 0x91117 (0x7f1d06026117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8778701Z frame #3: pthread_cond_wait + 0x211 (0x7f1d06028a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8779071Z frame #4: <unknown function> + 0x5a02023 (0x7f1c4a455023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8779641Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1c4a4591ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8780238Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1c4a4501d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8780617Z frame #7: <unknown function> + 0x79d655 (0x7f1c5b940655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8780929Z frame #8: <unknown function> + 0xdc253 (0x7f1c1298b253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8781136Z frame #9: <unknown function> + 0x94ac3 (0x7f1d06029ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8781355Z frame #10: <unknown function> + 0x126850 (0x7f1d060bb850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8781434Z 
2025-07-17T11:07:26.8781529Z SIGABRT(6), PID: 1822839, Thread 1822996: 
2025-07-17T11:07:26.8781998Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8782205Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8782482Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8782687Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8783195Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8783407Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8783621Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8783625Z 
2025-07-17T11:07:26.8783713Z SIGABRT(6), PID: 1822838, Thread 1823131: 
2025-07-17T11:07:26.8784183Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8784388Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8784595Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8784792Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8785288Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8785498Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8785776Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8785781Z 
2025-07-17T11:07:26.8785874Z SIGABRT(6), PID: 1822839, Thread 1823000: 
2025-07-17T11:07:26.8786343Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8786549Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8786755Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8786950Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8787447Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8787654Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8787861Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8787866Z 
2025-07-17T11:07:26.8787954Z SIGABRT(6), PID: 1822841, Thread 1827710: 
2025-07-17T11:07:26.8788428Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8788633Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8788907Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8789102Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8789477Z frame #4: <unknown function> + 0x5a02023 (0x7f7ff883f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8790112Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7ff88431ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8790758Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7ff883a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8791143Z frame #7: <unknown function> + 0x79d655 (0x7f8009d2a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8791375Z frame #8: <unknown function> + 0xdc253 (0x7f7fc0d75253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8791582Z frame #9: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8791794Z frame #10: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8791799Z 
2025-07-17T11:07:26.8791893Z SIGABRT(6), PID: 1822839, Thread 1823004: 
2025-07-17T11:07:26.8792358Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8792566Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8792768Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8792970Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8793477Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8793685Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8793957Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8793962Z 
2025-07-17T11:07:26.8794058Z SIGABRT(6), PID: 1822838, Thread 1823136: 
2025-07-17T11:07:26.8794527Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8794739Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8794943Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8795145Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8795647Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8795856Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8796066Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8796070Z 
2025-07-17T11:07:26.8796159Z SIGABRT(6), PID: 1822839, Thread 1823008: 
2025-07-17T11:07:26.8796626Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8796832Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8797105Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8797304Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8797796Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8798067Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8798274Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8798280Z 
2025-07-17T11:07:26.8798372Z SIGABRT(6), PID: 1822838, Thread 1823141: 
2025-07-17T11:07:26.8798916Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8799130Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8799336Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8799535Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8800029Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8800238Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8800447Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8800535Z SIGABRT(6), PID: 1822839, Thread 1823012: 
2025-07-17T11:07:26.8801004Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8801212Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8801413Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8801608Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8802167Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8802377Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8802585Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8802589Z 
2025-07-17T11:07:26.8802592Z 
2025-07-17T11:07:26.8802682Z SIGABRT(6), PID: 1822839, Thread 1823016: 
2025-07-17T11:07:26.8803151Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8803355Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8803558Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8803751Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8804247Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8804451Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8804659Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8804663Z 
2025-07-17T11:07:26.8804752Z SIGABRT(6), PID: 1822838, Thread 1823146: 
2025-07-17T11:07:26.8805298Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8805507Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8805712Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8805977Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8806466Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8806673Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8806941Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8806948Z 
2025-07-17T11:07:26.8807039Z SIGABRT(6), PID: 1822839, Thread 1823020: 
2025-07-17T11:07:26.8807508Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8807711Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8807917Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8808111Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8808605Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8808812Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8809020Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8809026Z 
2025-07-17T11:07:26.8809113Z SIGABRT(6), PID: 1822838, Thread 1823152: 
2025-07-17T11:07:26.8809583Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8809791Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8810056Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8810264Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8810756Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8810964Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8811172Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8811177Z 
2025-07-17T11:07:26.8811267Z SIGABRT(6), PID: 1822839, Thread 1823023: 
2025-07-17T11:07:26.8811731Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8811939Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8812137Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8812334Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8812828Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8813101Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8813308Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8813312Z 
2025-07-17T11:07:26.8813404Z SIGABRT(6), PID: 1822838, Thread 1823157: 
2025-07-17T11:07:26.8813868Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8814142Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8814346Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8814547Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8815106Z frame #4: <unknown function> + 0x3475ab (0x7ff719c085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8815320Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8815533Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8815537Z 
2025-07-17T11:07:26.8815629Z SIGABRT(6), PID: 1822841, Thread 1827712: 
2025-07-17T11:07:26.8816169Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f7fc2b08b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8816381Z frame #1: <unknown function> + 0x42520 (0x7f80b43c1520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8816587Z frame #2: <unknown function> + 0x91117 (0x7f80b4410117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8816788Z frame #3: pthread_cond_wait + 0x211 (0x7f80b4412a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8817163Z frame #4: <unknown function> + 0x5a02023 (0x7f7ff883f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8817731Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f7ff88431ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8818408Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f7ff883a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8818797Z frame #7: <unknown function> + 0x79d655 (0x7f8009d2a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8819028Z frame #8: <unknown function> + 0xdc253 (0x7f7fc0d75253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8819239Z frame #9: <unknown function> + 0x94ac3 (0x7f80b4413ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8819455Z frame #10: <unknown function> + 0x126850 (0x7f80b44a5850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8819463Z 
2025-07-17T11:07:26.8819552Z SIGABRT(6), PID: 1822838, Thread 1823898: 
2025-07-17T11:07:26.8820024Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8820475Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7ff62b46508a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8820685Z frame #2: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8820879Z frame #3: pthread_kill + 0x12c (0x7ff71cd729fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8821050Z frame #4: raise + 0x16 (0x7ff71cd1e476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8821211Z frame #5: abort + 0xd3 (0x7ff71cd047f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8821536Z frame #6: <unknown function> + 0x972cc (0x7ff598fee2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8821765Z frame #7: <unknown function> + 0x95f47 (0x7ff598fecf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8821991Z frame #8: <unknown function> + 0x34b71 (0x7ff598f8bb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8822201Z frame #9: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8822521Z frame #10: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8822525Z 
2025-07-17T11:07:26.8822622Z SIGABRT(6), PID: 1822839, Thread 1823027: 
2025-07-17T11:07:26.8823116Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8823416Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8823633Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8823839Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8824363Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8824574Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8824784Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8824788Z 
2025-07-17T11:07:26.8824879Z SIGABRT(6), PID: 1822838, Thread 1826616: 
2025-07-17T11:07:26.8825358Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8825567Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8825737Z frame #2: __poll + 0x4f (0x7ff71cdf4bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8826149Z frame #3: <unknown function> + 0x489f (0x7ff71c90089f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.8826383Z frame #4: <unknown function> + 0x1483bc (0x55a55c9763bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8826695Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8826931Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8827167Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8827392Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8827639Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55a55c95ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8827867Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8828114Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a55c95cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8828341Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8828582Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8828805Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8829040Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8829261Z frame #16: <unknown function> + 0x147734 (0x55a55c975734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8829481Z frame #17: <unknown function> + 0x21ca82 (0x55a55ca4aa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8829775Z frame #18: <unknown function> + 0x21c9e4 (0x55a55ca4a9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8829995Z frame #19: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8830209Z frame #20: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8830294Z 
2025-07-17T11:07:26.8830391Z SIGABRT(6), PID: 1822839, Thread 1823032: 
2025-07-17T11:07:26.8830876Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8831088Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8831353Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8831558Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8832071Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8832278Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8832488Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8832494Z 
2025-07-17T11:07:26.8832584Z SIGABRT(6), PID: 1822839, Thread 1823036: 
2025-07-17T11:07:26.8833052Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8833260Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8833473Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8833675Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8839195Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8839457Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8839804Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8839810Z 
2025-07-17T11:07:26.8839916Z SIGABRT(6), PID: 1822838, Thread 1827348: 
2025-07-17T11:07:26.8840437Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8840666Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8840883Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8841113Z frame #3: pthread_cond_clockwait + 0x1fd (0x7ff71cd702dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8841564Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7ff6571ad048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8841992Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7ff6571afbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8842231Z frame #6: <unknown function> + 0xdc253 (0x7ff6296d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8842443Z frame #7: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8842657Z frame #8: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8842662Z 
2025-07-17T11:07:26.8842838Z SIGABRT(6), PID: 1822839, Thread 1823040: 
2025-07-17T11:07:26.8843322Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8843529Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8843729Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8843997Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8844505Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8844775Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8844989Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8844996Z 
2025-07-17T11:07:26.8845092Z SIGABRT(6), PID: 1822838, Thread 1827349: 
2025-07-17T11:07:26.8845560Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8845770Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8845975Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8846203Z frame #3: pthread_cond_clockwait + 0x1fd (0x7ff71cd702dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8846671Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7ff6571a7741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.8846902Z frame #5: <unknown function> + 0xdc253 (0x7ff6296d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8847108Z frame #6: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8847317Z frame #7: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8847321Z 
2025-07-17T11:07:26.8847409Z SIGABRT(6), PID: 1822839, Thread 1823044: 
2025-07-17T11:07:26.8847870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8848139Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8848349Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8848544Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8849049Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8849253Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8849462Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8849467Z 
2025-07-17T11:07:26.8849553Z SIGABRT(6), PID: 1822838, Thread 1827350: 
2025-07-17T11:07:26.8850029Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8850234Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8850409Z frame #2: ioctl + 0x3f (0x7ff71cdf694f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8850641Z frame #3: <unknown function> + 0x13eef0 (0x7ff599095ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8850871Z frame #4: <unknown function> + 0x1378c4 (0x7ff59908e8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8851185Z frame #5: <unknown function> + 0x95b2b (0x7ff598fecb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8851414Z frame #6: <unknown function> + 0x34b71 (0x7ff598f8bb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8851623Z frame #7: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8851898Z frame #8: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8851902Z 
2025-07-17T11:07:26.8851994Z SIGABRT(6), PID: 1822839, Thread 1823048: 
2025-07-17T11:07:26.8852465Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8852733Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8852941Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8853140Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8853634Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8853837Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8854048Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8854052Z 
2025-07-17T11:07:26.8854141Z SIGABRT(6), PID: 1822838, Thread 1827649: 
2025-07-17T11:07:26.8854607Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8854820Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8854988Z frame #2: __poll + 0x4f (0x7ff71cdf4bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8855201Z frame #3: <unknown function> + 0xd4540 (0x7ff5e1205540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8855406Z frame #4: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8855614Z frame #5: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8855772Z SIGABRT(6), PID: 1822839, Thread 1823052: 
2025-07-17T11:07:26.8856344Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8856551Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8856755Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8856952Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8857448Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8857652Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8857861Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8857867Z 
2025-07-17T11:07:26.8857870Z 
2025-07-17T11:07:26.8857957Z SIGABRT(6), PID: 1822838, Thread 1827657: 
2025-07-17T11:07:26.8858420Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8858626Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8858791Z frame #2: __poll + 0x4f (0x7ff71cdf4bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8859094Z frame #3: <unknown function> + 0xd65a2 (0x7ff5e12075a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8859305Z frame #4: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8859510Z frame #5: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8859596Z 
2025-07-17T11:07:26.8859688Z SIGABRT(6), PID: 1822839, Thread 1823056: 
2025-07-17T11:07:26.8860152Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8860359Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8860632Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8860835Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8861333Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8861538Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8861745Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8861751Z 
2025-07-17T11:07:26.8861843Z SIGABRT(6), PID: 1822838, Thread 1827663: 
2025-07-17T11:07:26.8862303Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8862513Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8862715Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8862916Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8863127Z frame #4: <unknown function> + 0xd1e7f (0x7ff5e1202e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.8863335Z frame #5: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8863540Z frame #6: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8863546Z 
2025-07-17T11:07:26.8863829Z SIGABRT(6), PID: 1822839, Thread 1823060: 
2025-07-17T11:07:26.8864291Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8864498Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8864700Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8864899Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8865393Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8865596Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8865802Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8865806Z 
2025-07-17T11:07:26.8865899Z SIGABRT(6), PID: 1822838, Thread 1827676: 
2025-07-17T11:07:26.8866357Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8866567Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8866770Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8867067Z frame #3: <unknown function> + 0x9ca53 (0x7ff71cd78a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8867322Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55a55c942bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8867555Z frame #5: <unknown function> + 0x1d314f (0x55a55ca0114f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8867845Z frame #6: <unknown function> + 0x1d2f6b (0x55a55ca00f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8868065Z frame #7: <unknown function> + 0x6df36 (0x55a55c89bf36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8868307Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8868593Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8868835Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8869068Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8869303Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8869528Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8869761Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8869983Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8870213Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8870435Z frame #17: <unknown function> + 0x147734 (0x55a55c975734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8870652Z frame #18: <unknown function> + 0x21ca82 (0x55a55ca4aa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8870872Z frame #19: <unknown function> + 0x21c9e4 (0x55a55ca4a9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8871081Z frame #20: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8871291Z frame #21: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8871382Z SIGABRT(6), PID: 1822839, Thread 1823063: 
2025-07-17T11:07:26.8871918Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8872134Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8872338Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8872540Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8873044Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8873249Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8873454Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8873459Z 
2025-07-17T11:07:26.8873463Z 
2025-07-17T11:07:26.8873556Z SIGABRT(6), PID: 1822838, Thread 1827693: 
2025-07-17T11:07:26.8874022Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8874228Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8874392Z frame #2: read + 0x4c (0x7ff71cdf081c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8874645Z frame #3: _Py_read + 0x4c (0x55a55ca08a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8874869Z frame #4: <unknown function> + 0x208c3b (0x55a55ca36c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8875088Z frame #5: <unknown function> + 0x14648d (0x55a55c97448d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8875331Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55a55c983215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8875616Z frame #7: <unknown function> + 0x208ae7 (0x55a55ca36ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8875832Z frame #8: <unknown function> + 0x208472 (0x55a55ca36472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8876051Z frame #9: <unknown function> + 0x1da6ee (0x55a55ca086ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8876339Z frame #10: <unknown function> + 0x1483bc (0x55a55c9763bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8876585Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8876810Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8877046Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55a55c95a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8877263Z frame #14: <unknown function> + 0x147734 (0x55a55c975734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8877513Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55a55c95cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8877736Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8877968Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8878189Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55a55c96a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8878421Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55a55c95a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8878642Z frame #20: <unknown function> + 0x147734 (0x55a55c975734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8878861Z frame #21: <unknown function> + 0x21ca82 (0x55a55ca4aa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8879079Z frame #22: <unknown function> + 0x21c9e4 (0x55a55ca4a9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8879356Z frame #23: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8879572Z frame #24: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8879576Z 
2025-07-17T11:07:26.8879671Z SIGABRT(6), PID: 1822839, Thread 1823066: 
2025-07-17T11:07:26.8880141Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8880352Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8880559Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8880759Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8881253Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8881465Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8881669Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8881675Z 
2025-07-17T11:07:26.8881767Z SIGABRT(6), PID: 1822839, Thread 1823069: 
2025-07-17T11:07:26.8882231Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8882508Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8882707Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8882904Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8883405Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8883682Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8883888Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8883892Z 
2025-07-17T11:07:26.8883984Z SIGABRT(6), PID: 1822839, Thread 1823072: 
2025-07-17T11:07:26.8884500Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8884713Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8884912Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8885109Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8885606Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8885814Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8886018Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8886025Z 
2025-07-17T11:07:26.8886114Z SIGABRT(6), PID: 1822839, Thread 1823075: 
2025-07-17T11:07:26.8886576Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8886782Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8886991Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8887255Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8887758Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8887967Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8888174Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8888180Z 
2025-07-17T11:07:26.8888271Z SIGABRT(6), PID: 1822839, Thread 1823079: 
2025-07-17T11:07:26.8888740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8888945Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8889148Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8889346Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8889839Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8890043Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8890250Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8890320Z 
2025-07-17T11:07:26.8890411Z SIGABRT(6), PID: 1822838, Thread 1827746: 
2025-07-17T11:07:26.8890880Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8891085Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8891362Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8891560Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8891937Z frame #4: <unknown function> + 0x5a02023 (0x7ff66119b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8892565Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff66119f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8893164Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff6611961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8893544Z frame #7: <unknown function> + 0x79d655 (0x7ff672686655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8893778Z frame #8: <unknown function> + 0xdc253 (0x7ff6296d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8893985Z frame #9: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8894199Z frame #10: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8894203Z 
2025-07-17T11:07:26.8894293Z SIGABRT(6), PID: 1822839, Thread 1823082: 
2025-07-17T11:07:26.8894762Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8894967Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8895171Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8895367Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8896037Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8896257Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8896467Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8896474Z 
2025-07-17T11:07:26.8896564Z SIGABRT(6), PID: 1822839, Thread 1823085: 
2025-07-17T11:07:26.8897038Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8897239Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8897441Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8897638Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8898131Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8898336Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8898543Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8898627Z 
2025-07-17T11:07:26.8898719Z SIGABRT(6), PID: 1822839, Thread 1823088: 
2025-07-17T11:07:26.8899180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8899384Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8899661Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8899862Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8900352Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8900629Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8900838Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8900844Z 
2025-07-17T11:07:26.8900934Z SIGABRT(6), PID: 1822839, Thread 1823091: 
2025-07-17T11:07:26.8901393Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8901596Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8901798Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8901996Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8902485Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8902693Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8902900Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8902904Z 
2025-07-17T11:07:26.8902993Z SIGABRT(6), PID: 1822839, Thread 1823094: 
2025-07-17T11:07:26.8903453Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8903747Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8903951Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8904149Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8904637Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8904844Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8905047Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8905051Z 
2025-07-17T11:07:26.8905141Z SIGABRT(6), PID: 1822836, Thread 1822836: 
2025-07-17T11:07:26.8905600Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8905811Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8906040Z frame #2: <unknown function> + 0x77445 (0x7fdf74d4f445 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8906269Z frame #3: <unknown function> + 0x76c8e (0x7fdf74d4ec8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8906494Z frame #4: <unknown function> + 0x6ad69 (0x7fdf74d42d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8906777Z frame #5: <unknown function> + 0x3f54db (0x7fe0059a74db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8906986Z frame #6: <unknown function> + 0x3e11d8 (0x7fe0059931d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8907192Z frame #7: <unknown function> + 0xb2f86 (0x7fe005664f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8907395Z frame #8: <unknown function> + 0x8e501 (0x7fe005640501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8907842Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fe0304477ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.8908229Z frame #10: <unknown function> + 0xd307d9 (0x7fe04e99a7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8908458Z frame #11: <unknown function> + 0x139902 (0x56276fa1f902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8908767Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x56276fa16a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8909001Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8909242Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x56276fa16a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8909466Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8909706Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8909930Z frame #17: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8910163Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8910385Z frame #19: <unknown function> + 0x1474b2 (0x56276fa2d4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8910588Z frame #20: PyObject_Call + 0xbc (0x56276fa2de4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8910811Z frame #21: <unknown function> + 0x1ff69d (0x56276fae569d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8911032Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x56276fa1b023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8911274Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x56276fa16cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8911495Z frame #24: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8911800Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x56276fa123bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8912021Z frame #26: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8912255Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x56276fa123bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8912474Z frame #28: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8912715Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x56276fa16a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8912969Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x56276fa1a4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8913188Z frame #31: _PyObject_Call_Prepend + 0x69 (0x56276fa2bab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8913413Z frame #32: <unknown function> + 0x205949 (0x56276faeb949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8913640Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x56276fa1b023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8913877Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x56276fa16cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8914098Z frame #35: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8914332Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x56276fa123bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8914620Z frame #37: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8914857Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x56276fa123bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8915081Z frame #39: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8915314Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x56276fa123bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8915601Z frame #41: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8915839Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x56276fa133f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8916062Z frame #43: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8916510Z frame #44: <unknown function> + 0x7ab3f7 (0x7fe04e4153f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8916735Z frame #45: <unknown function> + 0x13bc88 (0x56276fa21c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8916939Z frame #46: PyObject_Call + 0x20f (0x56276fa2df9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8917182Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x56276fa1798d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8917400Z frame #48: <unknown function> + 0x147608 (0x56276fa2d608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8917642Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x56276fa14c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8917863Z frame #50: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8918097Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x56276fa123bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8918315Z frame #52: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8918552Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x56276fa133f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8918770Z frame #54: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8919002Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x56276fa123bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8919220Z frame #56: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8919526Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x56276fa14c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8919746Z frame #58: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8919985Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x56276fa14c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8920204Z frame #60: <unknown function> + 0x147608 (0x56276fa2d608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8920441Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x56276fa14c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8920656Z frame #62: <unknown function> + 0x147608 (0x56276fa2d608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8920749Z SIGABRT(6), PID: 1822839, Thread 1823097: 
2025-07-17T11:07:26.8921221Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8921440Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8921644Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8921845Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8922352Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8922630Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8922841Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8922845Z 
2025-07-17T11:07:26.8922851Z 
2025-07-17T11:07:26.8922941Z SIGABRT(6), PID: 1822838, Thread 1827748: 
2025-07-17T11:07:26.8923481Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8923688Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8923890Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8924170Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8924549Z frame #4: <unknown function> + 0x5a02023 (0x7ff66119b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8925115Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff66119f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8925704Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff6611961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8926083Z frame #7: <unknown function> + 0x79d655 (0x7ff672686655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8926313Z frame #8: <unknown function> + 0xdc253 (0x7ff6296d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8926521Z frame #9: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8926739Z frame #10: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8926743Z 
2025-07-17T11:07:26.8926833Z SIGABRT(6), PID: 1822839, Thread 1823100: 
2025-07-17T11:07:26.8927300Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8927575Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8927787Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8927985Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8928489Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8928694Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8928904Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8928908Z 
2025-07-17T11:07:26.8928996Z SIGABRT(6), PID: 1822836, Thread 1822852: 
2025-07-17T11:07:26.8929464Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8929670Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8929874Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8930071Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8930568Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8930847Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8931061Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8931065Z 
2025-07-17T11:07:26.8931155Z SIGABRT(6), PID: 1822839, Thread 1823103: 
2025-07-17T11:07:26.8931621Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8931886Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8932091Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8932286Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8932837Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8933047Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8933253Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8933257Z 
2025-07-17T11:07:26.8933347Z SIGABRT(6), PID: 1822836, Thread 1822853: 
2025-07-17T11:07:26.8933807Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8934014Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8934215Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8934414Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8934906Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8935114Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8935323Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8935329Z 
2025-07-17T11:07:26.8935419Z SIGABRT(6), PID: 1822839, Thread 1823106: 
2025-07-17T11:07:26.8936005Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8936219Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8936423Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8936622Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8937115Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8937320Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8937524Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8937530Z 
2025-07-17T11:07:26.8937622Z SIGABRT(6), PID: 1822836, Thread 1822854: 
2025-07-17T11:07:26.8938082Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8938288Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8938490Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8938770Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8939257Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8939466Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8939764Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8939769Z 
2025-07-17T11:07:26.8939859Z SIGABRT(6), PID: 1822839, Thread 1823109: 
2025-07-17T11:07:26.8940319Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8940599Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8940808Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8941010Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8941514Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8941724Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8941929Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8941932Z 
2025-07-17T11:07:26.8942022Z SIGABRT(6), PID: 1822836, Thread 1822856: 
2025-07-17T11:07:26.8942492Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8942702Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8942905Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8943110Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8943599Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8943891Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8944117Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8944126Z 
2025-07-17T11:07:26.8944220Z SIGABRT(6), PID: 1822839, Thread 1823114: 
2025-07-17T11:07:26.8944713Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8944927Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8945131Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8945328Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8945836Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8946045Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8946251Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8946255Z 
2025-07-17T11:07:26.8946343Z SIGABRT(6), PID: 1822836, Thread 1822857: 
2025-07-17T11:07:26.8946817Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8947109Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8947318Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8947518Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8948082Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8948290Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8948501Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8948505Z 
2025-07-17T11:07:26.8948654Z SIGABRT(6), PID: 1822838, Thread 1827750: 
2025-07-17T11:07:26.8949132Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8949344Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8949553Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8949756Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8950139Z frame #4: <unknown function> + 0x5a02023 (0x7ff66119b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8950708Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff66119f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8951308Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff6611961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8951690Z frame #7: <unknown function> + 0x79d655 (0x7ff672686655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8951923Z frame #8: <unknown function> + 0xdc253 (0x7ff6296d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8952201Z frame #9: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8952430Z frame #10: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8952435Z 
2025-07-17T11:07:26.8952525Z SIGABRT(6), PID: 1822842, Thread 1822842: 
2025-07-17T11:07:26.8953005Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8953211Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8953449Z frame #2: <unknown function> + 0x7743a (0x7f00eb1fb43a in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8953677Z frame #3: <unknown function> + 0x76c8e (0x7f00eb1fac8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8953900Z frame #4: <unknown function> + 0x6ad69 (0x7f00eb1eed69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.8954113Z frame #5: <unknown function> + 0x3f54db (0x7f017be534db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8954326Z frame #6: <unknown function> + 0x3e11d8 (0x7f017be3f1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8954534Z frame #7: <unknown function> + 0xb2f86 (0x7f017bb10f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8954740Z frame #8: <unknown function> + 0x8e501 (0x7f017baec501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.8955116Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f01a68f37ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.8955568Z frame #10: <unknown function> + 0xd307d9 (0x7f01c4e427d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8955798Z frame #11: <unknown function> + 0x139902 (0x562d2272c902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8956047Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x562d22723a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8956362Z frame #13: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8956601Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x562d22723a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8956825Z frame #15: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8957123Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8957350Z frame #17: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8957586Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8957809Z frame #19: <unknown function> + 0x1474b2 (0x562d2273a4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8958011Z frame #20: PyObject_Call + 0xbc (0x562d2273ae4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8958233Z frame #21: <unknown function> + 0x1ff69d (0x562d227f269d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8958456Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x562d22728023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8958694Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x562d22723cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8958922Z frame #24: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8959152Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x562d2271f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8959373Z frame #26: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8959602Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x562d2271f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8959823Z frame #28: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8960127Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x562d22723a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8960389Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x562d227274b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8960605Z frame #31: _PyObject_Call_Prepend + 0x69 (0x562d22738ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8960825Z frame #32: <unknown function> + 0x205949 (0x562d227f8949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8961045Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x562d22728023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8961289Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x562d22723cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8961510Z frame #35: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8961744Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x562d2271f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8961967Z frame #37: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8962200Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x562d2271f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8962418Z frame #39: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8962652Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x562d2271f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8962871Z frame #41: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8963183Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x562d227203f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8963404Z frame #43: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8963791Z frame #44: <unknown function> + 0x7ab3f7 (0x7f01c48bd3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8964071Z frame #45: <unknown function> + 0x13bc88 (0x562d2272ec88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8964275Z frame #46: PyObject_Call + 0x20f (0x562d2273af9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8964513Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x562d2272498d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8964795Z frame #48: <unknown function> + 0x147608 (0x562d2273a608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8965035Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x562d22721c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8965262Z frame #50: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8965494Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x562d2271f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8965718Z frame #52: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8965959Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x562d227203f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8966181Z frame #54: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8966411Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x562d2271f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8966633Z frame #56: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8966867Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x562d22721c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8967093Z frame #58: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8967328Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x562d22721c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8967543Z frame #60: <unknown function> + 0x147608 (0x562d2273a608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8967843Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x562d22721c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8968065Z frame #62: <unknown function> + 0x147608 (0x562d2273a608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.8968155Z SIGABRT(6), PID: 1822839, Thread 1823118: 
2025-07-17T11:07:26.8968635Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8968846Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8969060Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8969262Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8969772Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8969986Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8970194Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8970284Z SIGABRT(6), PID: 1822836, Thread 1822858: 
2025-07-17T11:07:26.8970759Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8971033Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8971235Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8971436Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8971938Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8972209Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8972416Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8972420Z 
2025-07-17T11:07:26.8972423Z 
2025-07-17T11:07:26.8972427Z 
2025-07-17T11:07:26.8972585Z SIGABRT(6), PID: 1822842, Thread 1823111: 
2025-07-17T11:07:26.8973057Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8973266Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8973467Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8973672Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8974168Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8974376Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8974584Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8974588Z 
2025-07-17T11:07:26.8974679Z SIGABRT(6), PID: 1822839, Thread 1823122: 
2025-07-17T11:07:26.8975143Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8975350Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8975549Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8975820Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8976411Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8976623Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8976847Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8976852Z 
2025-07-17T11:07:26.8976945Z SIGABRT(6), PID: 1822836, Thread 1822859: 
2025-07-17T11:07:26.8977412Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8977617Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8977822Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8978022Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8978514Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8978724Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8978929Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8979027Z 
2025-07-17T11:07:26.8979118Z SIGABRT(6), PID: 1822842, Thread 1823115: 
2025-07-17T11:07:26.8979588Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8979789Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8980071Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8980267Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8980762Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8981046Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8981259Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8981347Z SIGABRT(6), PID: 1822839, Thread 1823127: 
2025-07-17T11:07:26.8981813Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8982020Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8982221Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8982414Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8982909Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8983112Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8983322Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8983409Z SIGABRT(6), PID: 1822836, Thread 1822860: 
2025-07-17T11:07:26.8983878Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8984166Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8984376Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8984572Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8985068Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8985278Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8985486Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8985490Z 
2025-07-17T11:07:26.8985493Z 
2025-07-17T11:07:26.8985497Z 
2025-07-17T11:07:26.8985583Z SIGABRT(6), PID: 1822836, Thread 1822861: 
2025-07-17T11:07:26.8986053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8986258Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8986461Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8986661Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8987154Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8987456Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8987667Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8987755Z SIGABRT(6), PID: 1822839, Thread 1823132: 
2025-07-17T11:07:26.8988284Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8988488Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8988716Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8989099Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8989603Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8989812Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8990112Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8990118Z 
2025-07-17T11:07:26.8990121Z 
2025-07-17T11:07:26.8990238Z SIGABRT(6), PID: 1822842, Thread 1823119: 
2025-07-17T11:07:26.8990699Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8990905Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8991106Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8991304Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8991793Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8991999Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8992281Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8992286Z 
2025-07-17T11:07:26.8992377Z SIGABRT(6), PID: 1822838, Thread 1827751: 
2025-07-17T11:07:26.8992842Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8993051Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8993252Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8993459Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8993833Z frame #4: <unknown function> + 0x5a02023 (0x7ff66119b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8994401Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff66119f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8994998Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff6611961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.8995377Z frame #7: <unknown function> + 0x79d655 (0x7ff672686655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.8995608Z frame #8: <unknown function> + 0xdc253 (0x7ff6296d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.8995888Z frame #9: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8996102Z frame #10: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8996107Z 
2025-07-17T11:07:26.8996202Z SIGABRT(6), PID: 1822839, Thread 1823137: 
2025-07-17T11:07:26.8996735Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8996945Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8997146Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8997405Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8997905Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.8998118Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8998324Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8998418Z SIGABRT(6), PID: 1822842, Thread 1823123: 
2025-07-17T11:07:26.8998882Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.8999090Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8999288Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8999488Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.8999980Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9000188Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9000395Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9000488Z SIGABRT(6), PID: 1822836, Thread 1822862: 
2025-07-17T11:07:26.9001014Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9001225Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9001429Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9001628Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9002121Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9002326Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9002536Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9002542Z 
2025-07-17T11:07:26.9002547Z 
2025-07-17T11:07:26.9002550Z 
2025-07-17T11:07:26.9002638Z SIGABRT(6), PID: 1822839, Thread 1823142: 
2025-07-17T11:07:26.9003100Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9003305Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9003506Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9003770Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9004266Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9004468Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9004741Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9004745Z 
2025-07-17T11:07:26.9004833Z SIGABRT(6), PID: 1822836, Thread 1822863: 
2025-07-17T11:07:26.9005299Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9005563Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9005775Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9005974Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9006471Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9006681Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9006890Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9006894Z 
2025-07-17T11:07:26.9006982Z SIGABRT(6), PID: 1822842, Thread 1823128: 
2025-07-17T11:07:26.9007451Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9007652Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9007858Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9008054Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9008612Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9008822Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9009035Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9009039Z 
2025-07-17T11:07:26.9009127Z SIGABRT(6), PID: 1822839, Thread 1823147: 
2025-07-17T11:07:26.9009595Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9009801Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9010005Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9010197Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9010692Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9010898Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9011105Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9011109Z 
2025-07-17T11:07:26.9011200Z SIGABRT(6), PID: 1822842, Thread 1823134: 
2025-07-17T11:07:26.9011661Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9011936Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9012136Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9012331Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9012888Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9013097Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9013358Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9013363Z 
2025-07-17T11:07:26.9013455Z SIGABRT(6), PID: 1822836, Thread 1822864: 
2025-07-17T11:07:26.9013918Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9014126Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9014330Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9014534Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9015024Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9015233Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9015443Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9015448Z 
2025-07-17T11:07:26.9015538Z SIGABRT(6), PID: 1822839, Thread 1823151: 
2025-07-17T11:07:26.9016083Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9016294Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9016601Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9016800Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9017292Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9017500Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9017706Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9017712Z 
2025-07-17T11:07:26.9017803Z SIGABRT(6), PID: 1822842, Thread 1823138: 
2025-07-17T11:07:26.9018269Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9018473Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9018683Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9018881Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9019368Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9019577Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9019875Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9019881Z 
2025-07-17T11:07:26.9019973Z SIGABRT(6), PID: 1822836, Thread 1822865: 
2025-07-17T11:07:26.9020440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9020726Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9020931Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9021127Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9021691Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9021901Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9022112Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9022115Z 
2025-07-17T11:07:26.9022205Z SIGABRT(6), PID: 1822839, Thread 1823156: 
2025-07-17T11:07:26.9022669Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9022874Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9023076Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9023272Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9023764Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9023967Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9024172Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9024176Z 
2025-07-17T11:07:26.9024262Z SIGABRT(6), PID: 1822842, Thread 1823143: 
2025-07-17T11:07:26.9024792Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9025000Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9025206Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9025403Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9025897Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9026103Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9026310Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9026314Z 
2025-07-17T11:07:26.9026402Z SIGABRT(6), PID: 1822839, Thread 1823161: 
2025-07-17T11:07:26.9026870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9027072Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9027277Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9027471Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9028036Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9028240Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9028448Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9028514Z 
2025-07-17T11:07:26.9028610Z SIGABRT(6), PID: 1822838, Thread 1827753: 
2025-07-17T11:07:26.9029084Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9029290Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9029560Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9029762Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9030139Z frame #4: <unknown function> + 0x5a02023 (0x7ff66119b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9030698Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff66119f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9031296Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff6611961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9031673Z frame #7: <unknown function> + 0x79d655 (0x7ff672686655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9031902Z frame #8: <unknown function> + 0xdc253 (0x7ff6296d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9032115Z frame #9: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9032329Z frame #10: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9032333Z 
2025-07-17T11:07:26.9032425Z SIGABRT(6), PID: 1822836, Thread 1822866: 
2025-07-17T11:07:26.9032959Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9033174Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9033378Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9033579Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9034074Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9034284Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9034491Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9034495Z 
2025-07-17T11:07:26.9034587Z SIGABRT(6), PID: 1822842, Thread 1823149: 
2025-07-17T11:07:26.9035050Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9035254Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9035453Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9035652Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9036143Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9036419Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9036627Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9036690Z 
2025-07-17T11:07:26.9036782Z SIGABRT(6), PID: 1822839, Thread 1823165: 
2025-07-17T11:07:26.9037247Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9037452Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9037712Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9037915Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9038410Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9038617Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9038825Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9038830Z 
2025-07-17T11:07:26.9038923Z SIGABRT(6), PID: 1822836, Thread 1822867: 
2025-07-17T11:07:26.9039384Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9039590Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9039795Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9039995Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9040487Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9040695Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9040974Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9040982Z 
2025-07-17T11:07:26.9041073Z SIGABRT(6), PID: 1822839, Thread 1823169: 
2025-07-17T11:07:26.9041542Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9041749Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9041952Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9042148Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9042642Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9042846Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9043055Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9043059Z 
2025-07-17T11:07:26.9043146Z SIGABRT(6), PID: 1822842, Thread 1823154: 
2025-07-17T11:07:26.9043612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9043813Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9044091Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9044288Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9044777Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9045048Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9045256Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9045260Z 
2025-07-17T11:07:26.9045348Z SIGABRT(6), PID: 1822839, Thread 1823173: 
2025-07-17T11:07:26.9045897Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9046107Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9046312Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9046506Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9047004Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9047208Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9047416Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9047420Z 
2025-07-17T11:07:26.9047508Z SIGABRT(6), PID: 1822836, Thread 1822868: 
2025-07-17T11:07:26.9047976Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9048182Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9048386Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9048583Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9049143Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9049352Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9049560Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9049564Z 
2025-07-17T11:07:26.9049653Z SIGABRT(6), PID: 1822842, Thread 1823159: 
2025-07-17T11:07:26.9050117Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9050323Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9050525Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9050718Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9051216Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9051422Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9051632Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9051636Z 
2025-07-17T11:07:26.9051724Z SIGABRT(6), PID: 1822839, Thread 1823177: 
2025-07-17T11:07:26.9052254Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9052461Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9052660Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9052921Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9053415Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9053626Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9053893Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9053900Z 
2025-07-17T11:07:26.9053992Z SIGABRT(6), PID: 1822836, Thread 1822869: 
2025-07-17T11:07:26.9054451Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9054657Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9054862Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9055060Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9055550Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9055758Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9056033Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9056039Z 
2025-07-17T11:07:26.9056130Z SIGABRT(6), PID: 1822842, Thread 1823162: 
2025-07-17T11:07:26.9056590Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9056795Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9057078Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9057282Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9057773Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9057980Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9058188Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9058192Z 
2025-07-17T11:07:26.9058283Z SIGABRT(6), PID: 1822839, Thread 1823181: 
2025-07-17T11:07:26.9058741Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9058952Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9059151Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9059346Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9059838Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9060124Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9060328Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9060332Z 
2025-07-17T11:07:26.9060423Z SIGABRT(6), PID: 1822836, Thread 1822870: 
2025-07-17T11:07:26.9060886Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9061174Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9061375Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9061574Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9062133Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9062347Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9062555Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9062562Z 
2025-07-17T11:07:26.9062650Z SIGABRT(6), PID: 1822839, Thread 1823183: 
2025-07-17T11:07:26.9063118Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9063320Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9063522Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9063717Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9064210Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9064415Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9064624Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9064628Z 
2025-07-17T11:07:26.9064716Z SIGABRT(6), PID: 1822842, Thread 1823166: 
2025-07-17T11:07:26.9065251Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9065458Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9065661Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9065856Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9066347Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9066551Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9066760Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9066766Z 
2025-07-17T11:07:26.9066854Z SIGABRT(6), PID: 1822836, Thread 1822871: 
2025-07-17T11:07:26.9067321Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9067525Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9067730Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9067993Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9068485Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9068689Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9068960Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9068964Z 
2025-07-17T11:07:26.9069052Z SIGABRT(6), PID: 1822839, Thread 1823187: 
2025-07-17T11:07:26.9069520Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9069788Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9069997Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9070194Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9070689Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9070891Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9071100Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9071104Z 
2025-07-17T11:07:26.9071193Z SIGABRT(6), PID: 1822842, Thread 1823170: 
2025-07-17T11:07:26.9071659Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9071862Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9072065Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9072257Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9072746Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9073020Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9073230Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9073234Z 
2025-07-17T11:07:26.9073325Z SIGABRT(6), PID: 1822838, Thread 1827754: 
2025-07-17T11:07:26.9073786Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9073992Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9074196Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9074393Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9074765Z frame #4: <unknown function> + 0x5a02023 (0x7ff66119b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9075329Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff66119f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9075918Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff6611961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9076296Z frame #7: <unknown function> + 0x79d655 (0x7ff672686655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9076596Z frame #8: <unknown function> + 0xdc253 (0x7ff6296d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9076806Z frame #9: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9077021Z frame #10: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9077107Z 
2025-07-17T11:07:26.9077201Z SIGABRT(6), PID: 1822839, Thread 1823191: 
2025-07-17T11:07:26.9077668Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9077875Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9078134Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9078339Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9078833Z frame #4: <unknown function> + 0x3475ab (0x7f6cfff095ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9079043Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9079252Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9079256Z 
2025-07-17T11:07:26.9079348Z SIGABRT(6), PID: 1822836, Thread 1822872: 
2025-07-17T11:07:26.9079813Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9080022Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9080222Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9080424Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9080919Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9081130Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9081402Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9081407Z 
2025-07-17T11:07:26.9081501Z SIGABRT(6), PID: 1822842, Thread 1823174: 
2025-07-17T11:07:26.9081967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9082173Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9082375Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9082574Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9083066Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9083274Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9083485Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9083489Z 
2025-07-17T11:07:26.9083577Z SIGABRT(6), PID: 1822839, Thread 1823894: 
2025-07-17T11:07:26.9084044Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9084558Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f6c1176908a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9084766Z frame #2: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9084958Z frame #3: pthread_kill + 0x12c (0x7f6d030739fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9085184Z frame #4: raise + 0x16 (0x7f6d0301f476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9085344Z frame #5: abort + 0xd3 (0x7f6d030057f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9085585Z frame #6: <unknown function> + 0x972cc (0x7f6b7f2f22cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9085811Z frame #7: <unknown function> + 0x95f47 (0x7f6b7f2f0f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9086095Z frame #8: <unknown function> + 0x34b71 (0x7f6b7f28fb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9086307Z frame #9: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9086523Z frame #10: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9086528Z 
2025-07-17T11:07:26.9086617Z SIGABRT(6), PID: 1822842, Thread 1823180: 
2025-07-17T11:07:26.9087088Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9087295Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9087499Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9087698Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9088191Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9088396Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9088609Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9088698Z SIGABRT(6), PID: 1822836, Thread 1822873: 
2025-07-17T11:07:26.9089232Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9089442Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9089650Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9089846Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9090343Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9090552Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9090758Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9090762Z 
2025-07-17T11:07:26.9090766Z 
2025-07-17T11:07:26.9090853Z SIGABRT(6), PID: 1822839, Thread 1826417: 
2025-07-17T11:07:26.9091324Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9091528Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9091693Z frame #2: __poll + 0x4f (0x7f6d030f5bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9092098Z frame #3: <unknown function> + 0x489f (0x7f6d02c0189f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.9092394Z frame #4: <unknown function> + 0x1483bc (0x55b30f52b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9092637Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9092867Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9093164Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9093389Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9093632Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55b30f513a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9093865Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9094168Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b30f511c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9094401Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9094635Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9094860Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9095090Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9095316Z frame #16: <unknown function> + 0x147734 (0x55b30f52a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9095535Z frame #17: <unknown function> + 0x21ca82 (0x55b30f5ffa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9095755Z frame #18: <unknown function> + 0x21c9e4 (0x55b30f5ff9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9096034Z frame #19: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9096251Z frame #20: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9096255Z 
2025-07-17T11:07:26.9096344Z SIGABRT(6), PID: 1822836, Thread 1822874: 
2025-07-17T11:07:26.9096816Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9097109Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9097323Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9097525Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9098028Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9098235Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9098449Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9098538Z SIGABRT(6), PID: 1822842, Thread 1823185: 
2025-07-17T11:07:26.9099003Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9099211Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9099415Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9099610Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9100105Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9100400Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9100610Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9100615Z 
2025-07-17T11:07:26.9100618Z 
2025-07-17T11:07:26.9100709Z SIGABRT(6), PID: 1822842, Thread 1823189: 
2025-07-17T11:07:26.9101170Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9101454Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9101655Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9101852Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9102414Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9102626Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9102832Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9102926Z SIGABRT(6), PID: 1822836, Thread 1822875: 
2025-07-17T11:07:26.9103391Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9103597Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9103798Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9103997Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9104485Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9104693Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9104900Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9104906Z 
2025-07-17T11:07:26.9104909Z 
2025-07-17T11:07:26.9104999Z SIGABRT(6), PID: 1822842, Thread 1823193: 
2025-07-17T11:07:26.9105522Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9105731Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9105933Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9106132Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9106621Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9106830Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9107034Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9107041Z 
2025-07-17T11:07:26.9107131Z SIGABRT(6), PID: 1822839, Thread 1827420: 
2025-07-17T11:07:26.9107596Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9107803Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9108005Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9108324Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6d030712dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9108767Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f6c3d4b1048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9109190Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f6c3d4b3bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9109486Z frame #6: <unknown function> + 0xdc253 (0x7f6c0f9d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9109693Z frame #7: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9109896Z frame #8: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9109905Z 
2025-07-17T11:07:26.9110058Z SIGABRT(6), PID: 1822838, Thread 1827756: 
2025-07-17T11:07:26.9110528Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9110740Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9110949Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9111145Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9111524Z frame #4: <unknown function> + 0x5a02023 (0x7ff66119b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9112090Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff66119f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9112687Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff6611961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9113066Z frame #7: <unknown function> + 0x79d655 (0x7ff672686655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9113297Z frame #8: <unknown function> + 0xdc253 (0x7ff6296d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9113570Z frame #9: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9113792Z frame #10: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9113797Z 
2025-07-17T11:07:26.9113885Z SIGABRT(6), PID: 1822836, Thread 1822877: 
2025-07-17T11:07:26.9114357Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9114562Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9114771Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9114969Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9115467Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9115676Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9115889Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9115893Z 
2025-07-17T11:07:26.9115982Z SIGABRT(6), PID: 1822842, Thread 1823196: 
2025-07-17T11:07:26.9116448Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9116728Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9116931Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9117126Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9117623Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9117991Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9118200Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9118204Z 
2025-07-17T11:07:26.9118291Z SIGABRT(6), PID: 1822836, Thread 1822878: 
2025-07-17T11:07:26.9118817Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9119028Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9119237Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9119432Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9119932Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9120139Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9120349Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9120355Z 
2025-07-17T11:07:26.9120445Z SIGABRT(6), PID: 1822842, Thread 1823199: 
2025-07-17T11:07:26.9120912Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9121117Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9121317Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9121577Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9122071Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9122279Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9122487Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9122491Z 
2025-07-17T11:07:26.9122582Z SIGABRT(6), PID: 1822839, Thread 1827421: 
2025-07-17T11:07:26.9123042Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9123244Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9123446Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9123673Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6d030712dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9124143Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f6c3d4ab741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9124374Z frame #5: <unknown function> + 0xdc253 (0x7f6c0f9d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9124577Z frame #6: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9124852Z frame #7: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9124856Z 
2025-07-17T11:07:26.9124944Z SIGABRT(6), PID: 1822839, Thread 1827422: 
2025-07-17T11:07:26.9125405Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9125675Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9125844Z frame #2: ioctl + 0x3f (0x7f6d030f794f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9126078Z frame #3: <unknown function> + 0x13eef0 (0x7f6b7f399ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9126309Z frame #4: <unknown function> + 0x1378c4 (0x7f6b7f3928c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9126596Z frame #5: <unknown function> + 0x95b2b (0x7f6b7f2f0b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9126830Z frame #6: <unknown function> + 0x34b71 (0x7f6b7f28fb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9127041Z frame #7: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9127249Z frame #8: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9127253Z 
2025-07-17T11:07:26.9127343Z SIGABRT(6), PID: 1822836, Thread 1822880: 
2025-07-17T11:07:26.9127811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9128017Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9128221Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9128421Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9128920Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9129126Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9129336Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9129491Z SIGABRT(6), PID: 1822842, Thread 1823202: 
2025-07-17T11:07:26.9129959Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9130164Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9130370Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9130566Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9131066Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9131271Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9131483Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9131489Z 
2025-07-17T11:07:26.9131493Z 
2025-07-17T11:07:26.9131579Z SIGABRT(6), PID: 1822836, Thread 1822881: 
2025-07-17T11:07:26.9132044Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9132252Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9132454Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9132725Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9133216Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9133426Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9133703Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9133802Z SIGABRT(6), PID: 1822842, Thread 1823205: 
2025-07-17T11:07:26.9134262Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9134524Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9134733Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9134932Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9135419Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9135628Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9135833Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9135995Z SIGABRT(6), PID: 1822839, Thread 1827651: 
2025-07-17T11:07:26.9136460Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9136665Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9136834Z frame #2: __poll + 0x4f (0x7f6d030f5bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9137047Z frame #3: <unknown function> + 0xd4540 (0x7f6bc7509540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9137260Z frame #4: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9137471Z frame #5: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9137583Z 
2025-07-17T11:07:26.9137587Z 
2025-07-17T11:07:26.9137590Z 
2025-07-17T11:07:26.9137682Z SIGABRT(6), PID: 1822839, Thread 1827659: 
2025-07-17T11:07:26.9138160Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9138369Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9138533Z frame #2: __poll + 0x4f (0x7f6d030f5bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9138748Z frame #3: <unknown function> + 0xd65a2 (0x7f6bc750b5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9138957Z frame #4: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9139161Z frame #5: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9139167Z 
2025-07-17T11:07:26.9139258Z SIGABRT(6), PID: 1822842, Thread 1823208: 
2025-07-17T11:07:26.9139735Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9139956Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9140160Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9140362Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9140954Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9141165Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9141371Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9141445Z 
2025-07-17T11:07:26.9141541Z SIGABRT(6), PID: 1822836, Thread 1822882: 
2025-07-17T11:07:26.9142017Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9142227Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9142504Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9142710Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9143206Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9143415Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9143626Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9143632Z 
2025-07-17T11:07:26.9143721Z SIGABRT(6), PID: 1822839, Thread 1827665: 
2025-07-17T11:07:26.9144196Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9144401Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9144608Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9144801Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9145016Z frame #4: <unknown function> + 0xd1e7f (0x7f6bc7506e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9145220Z frame #5: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9145492Z frame #6: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9145497Z 
2025-07-17T11:07:26.9145589Z SIGABRT(6), PID: 1822842, Thread 1823211: 
2025-07-17T11:07:26.9146055Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9146260Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9146463Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9146661Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9147155Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9147365Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9147575Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9147579Z 
2025-07-17T11:07:26.9147667Z SIGABRT(6), PID: 1822836, Thread 1822884: 
2025-07-17T11:07:26.9148136Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9148342Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9148623Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9148823Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9149316Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9149592Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9149805Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9149810Z 
2025-07-17T11:07:26.9149897Z SIGABRT(6), PID: 1822838, Thread 1827757: 
2025-07-17T11:07:26.9150424Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7ff62b464b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9150636Z frame #1: <unknown function> + 0x42520 (0x7ff71cd1e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9150842Z frame #2: <unknown function> + 0x91117 (0x7ff71cd6d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9151038Z frame #3: pthread_cond_wait + 0x211 (0x7ff71cd6fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9151417Z frame #4: <unknown function> + 0x5a02023 (0x7ff66119b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9151987Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7ff66119f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9152582Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7ff6611961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9152963Z frame #7: <unknown function> + 0x79d655 (0x7ff672686655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9153199Z frame #8: <unknown function> + 0xdc253 (0x7ff6296d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9153404Z frame #9: <unknown function> + 0x94ac3 (0x7ff71cd70ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9153622Z frame #10: <unknown function> + 0x126850 (0x7ff71ce02850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9153697Z 
2025-07-17T11:07:26.9153791Z SIGABRT(6), PID: 1822839, Thread 1827675: 
2025-07-17T11:07:26.9154259Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9154470Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9154676Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9154888Z frame #3: <unknown function> + 0x9ca53 (0x7f6d03079a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9155141Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55b30f4f7bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9155370Z frame #5: <unknown function> + 0x1d314f (0x55b30f5b614f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9155593Z frame #6: <unknown function> + 0x1d2f6b (0x55b30f5b5f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9155815Z frame #7: <unknown function> + 0x6df36 (0x55b30f450f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9156056Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9156280Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9156520Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9156817Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9157050Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9157274Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9157504Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9157793Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9158024Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9158245Z frame #17: <unknown function> + 0x147734 (0x55b30f52a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9158526Z frame #18: <unknown function> + 0x21ca82 (0x55b30f5ffa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9158752Z frame #19: <unknown function> + 0x21c9e4 (0x55b30f5ff9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9158965Z frame #20: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9159175Z frame #21: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9159180Z 
2025-07-17T11:07:26.9159268Z SIGABRT(6), PID: 1822842, Thread 1823214: 
2025-07-17T11:07:26.9159745Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9159955Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9160160Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9160361Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9160867Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9161074Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9161281Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9161287Z 
2025-07-17T11:07:26.9161437Z SIGABRT(6), PID: 1822836, Thread 1822886: 
2025-07-17T11:07:26.9161911Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9162116Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9162324Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9162521Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9163023Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9163229Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9163438Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9163445Z 
2025-07-17T11:07:26.9163532Z SIGABRT(6), PID: 1822842, Thread 1823217: 
2025-07-17T11:07:26.9163999Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9164201Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9164409Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9164675Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9165167Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9165372Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9165649Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9165653Z 
2025-07-17T11:07:26.9165744Z SIGABRT(6), PID: 1822839, Thread 1827689: 
2025-07-17T11:07:26.9166212Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9166493Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9166670Z frame #2: read + 0x4c (0x7f6d030f181c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9166850Z frame #3: _Py_read + 0x4c (0x55b30f5bda6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9167078Z frame #4: <unknown function> + 0x208c3b (0x55b30f5ebc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9167297Z frame #5: <unknown function> + 0x14648d (0x55b30f52948d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9167545Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55b30f538215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9167765Z frame #7: <unknown function> + 0x208ae7 (0x55b30f5ebae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9167983Z frame #8: <unknown function> + 0x208472 (0x55b30f5eb472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9168204Z frame #9: <unknown function> + 0x1da6ee (0x55b30f5bd6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9168428Z frame #10: <unknown function> + 0x1483bc (0x55b30f52b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9168669Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9168896Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9169128Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55b30f50f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9169414Z frame #14: <unknown function> + 0x147734 (0x55b30f52a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9169665Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55b30f511c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9169893Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9170125Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9170353Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55b30f51f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9170585Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55b30f50f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9170811Z frame #20: <unknown function> + 0x147734 (0x55b30f52a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9171029Z frame #21: <unknown function> + 0x21ca82 (0x55b30f5ffa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9171252Z frame #22: <unknown function> + 0x21c9e4 (0x55b30f5ff9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9171464Z frame #23: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9171674Z frame #24: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9171678Z 
2025-07-17T11:07:26.9171765Z SIGABRT(6), PID: 1822836, Thread 1822888: 
2025-07-17T11:07:26.9172240Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9172523Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9172734Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9172932Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9173495Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9173703Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9173913Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9174002Z SIGABRT(6), PID: 1822842, Thread 1823220: 
2025-07-17T11:07:26.9174540Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9174752Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9174965Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9175162Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9175661Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9175869Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9176159Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9176166Z 
2025-07-17T11:07:26.9176170Z 
2025-07-17T11:07:26.9176262Z SIGABRT(6), PID: 1822842, Thread 1823222: 
2025-07-17T11:07:26.9176725Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9176935Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9177138Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9177422Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9177916Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9178125Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9178332Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9178338Z 
2025-07-17T11:07:26.9178429Z SIGABRT(6), PID: 1822836, Thread 1822891: 
2025-07-17T11:07:26.9178891Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9179098Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9179302Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9179501Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9179991Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9180205Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9180411Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9180492Z 
2025-07-17T11:07:26.9180584Z SIGABRT(6), PID: 1822842, Thread 1823224: 
2025-07-17T11:07:26.9181044Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9181250Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9181530Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9181729Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9182223Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9182508Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9182721Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9182725Z 
2025-07-17T11:07:26.9182816Z SIGABRT(6), PID: 1822836, Thread 1822892: 
2025-07-17T11:07:26.9183275Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9183484Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9183687Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9183885Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9184383Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9184593Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9184800Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9184806Z 
2025-07-17T11:07:26.9184892Z SIGABRT(6), PID: 1822836, Thread 1822895: 
2025-07-17T11:07:26.9185359Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9185633Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9185843Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9186039Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9186534Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9186742Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9186954Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9186958Z 
2025-07-17T11:07:26.9187044Z SIGABRT(6), PID: 1822842, Thread 1823226: 
2025-07-17T11:07:26.9187512Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9187717Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9187920Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9188115Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9188607Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9188883Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9189092Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9189096Z 
2025-07-17T11:07:26.9189248Z SIGABRT(6), PID: 1822842, Thread 1823228: 
2025-07-17T11:07:26.9189715Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9189920Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9190126Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9190384Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9190886Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9191092Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9191303Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9191309Z 
2025-07-17T11:07:26.9191401Z SIGABRT(6), PID: 1822836, Thread 1822896: 
2025-07-17T11:07:26.9191873Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9192077Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9192285Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9192483Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9192976Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9193181Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9193394Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9193465Z 
2025-07-17T11:07:26.9193556Z SIGABRT(6), PID: 1822842, Thread 1823230: 
2025-07-17T11:07:26.9194025Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9194231Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9194435Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9194631Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9195124Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9195329Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9195541Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9195545Z 
2025-07-17T11:07:26.9195631Z SIGABRT(6), PID: 1822836, Thread 1822898: 
2025-07-17T11:07:26.9196098Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9196308Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9196586Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9196785Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9197276Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9197571Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9197778Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9197782Z 
2025-07-17T11:07:26.9197871Z SIGABRT(6), PID: 1822839, Thread 1827734: 
2025-07-17T11:07:26.9198399Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9198610Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9198815Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9199014Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9199388Z frame #4: <unknown function> + 0x5a02023 (0x7f6c4749f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9199963Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c474a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9200557Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c4749a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9206369Z frame #7: <unknown function> + 0x79d655 (0x7f6c5898a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9206669Z frame #8: <unknown function> + 0xdc253 (0x7f6c0f9d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9206900Z frame #9: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9207123Z frame #10: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9207131Z 
2025-07-17T11:07:26.9207345Z SIGABRT(6), PID: 1822842, Thread 1823233: 
2025-07-17T11:07:26.9207860Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9208090Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9208301Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9208518Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9209037Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9209252Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9209473Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9209478Z 
2025-07-17T11:07:26.9209576Z SIGABRT(6), PID: 1822836, Thread 1822901: 
2025-07-17T11:07:26.9210062Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9210288Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9210497Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9210780Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9211297Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9211510Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9211789Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9211797Z 
2025-07-17T11:07:26.9211890Z SIGABRT(6), PID: 1822842, Thread 1823235: 
2025-07-17T11:07:26.9212370Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9212643Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9212854Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9213056Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9213563Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9213775Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9213986Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9213991Z 
2025-07-17T11:07:26.9214080Z SIGABRT(6), PID: 1822836, Thread 1822903: 
2025-07-17T11:07:26.9214562Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9214768Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9214975Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9215172Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9215727Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9216026Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9216240Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9216245Z 
2025-07-17T11:07:26.9216334Z SIGABRT(6), PID: 1822842, Thread 1823237: 
2025-07-17T11:07:26.9216803Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9217009Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9217212Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9217410Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9217907Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9218113Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9218321Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9218326Z 
2025-07-17T11:07:26.9218415Z SIGABRT(6), PID: 1822836, Thread 1822905: 
2025-07-17T11:07:26.9218882Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9219183Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9219388Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9219584Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9220160Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9220369Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9220652Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9220657Z 
2025-07-17T11:07:26.9220750Z SIGABRT(6), PID: 1822842, Thread 1823238: 
2025-07-17T11:07:26.9221227Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9221429Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9221634Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9221833Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9222331Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9222538Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9222747Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9222752Z 
2025-07-17T11:07:26.9222846Z SIGABRT(6), PID: 1822836, Thread 1822907: 
2025-07-17T11:07:26.9223311Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9223516Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9223720Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9223995Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9224488Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9224699Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9224908Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9224915Z 
2025-07-17T11:07:26.9225005Z SIGABRT(6), PID: 1822842, Thread 1823240: 
2025-07-17T11:07:26.9225472Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9225677Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9225880Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9226077Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9226568Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9226778Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9227054Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9227057Z 
2025-07-17T11:07:26.9227148Z SIGABRT(6), PID: 1822836, Thread 1822909: 
2025-07-17T11:07:26.9227613Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9227884Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9228086Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9228286Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9228840Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9229055Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9229262Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9229266Z 
2025-07-17T11:07:26.9229357Z SIGABRT(6), PID: 1822842, Thread 1823242: 
2025-07-17T11:07:26.9229822Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9230033Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9230233Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9230430Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9230924Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9231136Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9231343Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9231347Z 
2025-07-17T11:07:26.9231437Z SIGABRT(6), PID: 1822836, Thread 1822911: 
2025-07-17T11:07:26.9231969Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9232180Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9232381Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9232581Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9233069Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9233281Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9233489Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9233496Z 
2025-07-17T11:07:26.9233585Z SIGABRT(6), PID: 1822839, Thread 1827735: 
2025-07-17T11:07:26.9234052Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9234258Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9234461Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9234656Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9235126Z frame #4: <unknown function> + 0x5a02023 (0x7f6c4749f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9235696Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c474a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9236356Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c4749a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9236736Z frame #7: <unknown function> + 0x79d655 (0x7f6c5898a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9237031Z frame #8: <unknown function> + 0xdc253 (0x7f6c0f9d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9237243Z frame #9: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9237464Z frame #10: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9237554Z SIGABRT(6), PID: 1822842, Thread 1823244: 
2025-07-17T11:07:26.9238025Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9238235Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9238439Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9238635Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9239139Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9239346Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9239556Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9239561Z 
2025-07-17T11:07:26.9239564Z 
2025-07-17T11:07:26.9239653Z SIGABRT(6), PID: 1822842, Thread 1823246: 
2025-07-17T11:07:26.9240184Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9240395Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9240600Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9240801Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9241309Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9241515Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9241723Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9241728Z 
2025-07-17T11:07:26.9241815Z SIGABRT(6), PID: 1822836, Thread 1822913: 
2025-07-17T11:07:26.9242286Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9242489Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9242695Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9242894Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9243389Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9243666Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9243874Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9243940Z 
2025-07-17T11:07:26.9244033Z SIGABRT(6), PID: 1822842, Thread 1823248: 
2025-07-17T11:07:26.9244504Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9244710Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9244973Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9245175Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9245669Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9245876Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9246082Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9246087Z 
2025-07-17T11:07:26.9246179Z SIGABRT(6), PID: 1822836, Thread 1822915: 
2025-07-17T11:07:26.9246645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9246850Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9247053Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9247254Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9247745Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9247952Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9248222Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9248227Z 
2025-07-17T11:07:26.9248319Z SIGABRT(6), PID: 1822842, Thread 1823250: 
2025-07-17T11:07:26.9248785Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9248993Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9249194Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9249391Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9249883Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9250089Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9250297Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9250301Z 
2025-07-17T11:07:26.9250391Z SIGABRT(6), PID: 1822836, Thread 1822916: 
2025-07-17T11:07:26.9250853Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9251060Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9251433Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9251634Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9252122Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9252400Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9252607Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9252611Z 
2025-07-17T11:07:26.9252703Z SIGABRT(6), PID: 1822842, Thread 1823252: 
2025-07-17T11:07:26.9253226Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9253438Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9253641Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9253839Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9254333Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9254541Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9254748Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9254752Z 
2025-07-17T11:07:26.9254839Z SIGABRT(6), PID: 1822836, Thread 1822918: 
2025-07-17T11:07:26.9255311Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9255515Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9255719Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9255985Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9256568Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9256779Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9256988Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9256993Z 
2025-07-17T11:07:26.9257085Z SIGABRT(6), PID: 1822842, Thread 1823254: 
2025-07-17T11:07:26.9257556Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9257759Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9257961Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9258156Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9258654Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9258857Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9259068Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9259072Z 
2025-07-17T11:07:26.9259161Z SIGABRT(6), PID: 1822836, Thread 1822920: 
2025-07-17T11:07:26.9259707Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9259911Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9260118Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9260398Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9260894Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9261099Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9261385Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9261392Z 
2025-07-17T11:07:26.9261482Z SIGABRT(6), PID: 1822842, Thread 1823256: 
2025-07-17T11:07:26.9261959Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9262163Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9262367Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9262564Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9263060Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9263269Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9263477Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9263482Z 
2025-07-17T11:07:26.9263571Z SIGABRT(6), PID: 1822839, Thread 1827736: 
2025-07-17T11:07:26.9264037Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9264242Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9264540Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9264742Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9265117Z frame #4: <unknown function> + 0x5a02023 (0x7f6c4749f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9265691Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c474a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9266283Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c4749a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9266669Z frame #7: <unknown function> + 0x79d655 (0x7f6c5898a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9266907Z frame #8: <unknown function> + 0xdc253 (0x7f6c0f9d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9267114Z frame #9: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9267326Z frame #10: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9267330Z 
2025-07-17T11:07:26.9267424Z SIGABRT(6), PID: 1822836, Thread 1822923: 
2025-07-17T11:07:26.9267890Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9268168Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9268373Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9268646Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9269151Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9269362Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9269632Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9269636Z 
2025-07-17T11:07:26.9269740Z SIGABRT(6), PID: 1822842, Thread 1823258: 
2025-07-17T11:07:26.9270210Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9270418Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9270620Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9270821Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9271316Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9271522Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9271729Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9271735Z 
2025-07-17T11:07:26.9271829Z SIGABRT(6), PID: 1822836, Thread 1822925: 
2025-07-17T11:07:26.9272295Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9272502Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9272769Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9272972Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9273468Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9273679Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9273889Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9273898Z 
2025-07-17T11:07:26.9273986Z SIGABRT(6), PID: 1822842, Thread 1823260: 
2025-07-17T11:07:26.9274457Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9274661Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9274868Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9275062Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9275555Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9275760Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9276039Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9276043Z 
2025-07-17T11:07:26.9276131Z SIGABRT(6), PID: 1822836, Thread 1822927: 
2025-07-17T11:07:26.9276597Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9276870Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9277076Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9277274Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9277830Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9278041Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9278251Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9278255Z 
2025-07-17T11:07:26.9278343Z SIGABRT(6), PID: 1822842, Thread 1823262: 
2025-07-17T11:07:26.9278811Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9279015Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9279220Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9279415Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9279910Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9280118Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9280326Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9280330Z 
2025-07-17T11:07:26.9280417Z SIGABRT(6), PID: 1822836, Thread 1822929: 
2025-07-17T11:07:26.9280950Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9281158Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9281367Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9281565Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9282059Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9282269Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9282479Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9282484Z 
2025-07-17T11:07:26.9282571Z SIGABRT(6), PID: 1822842, Thread 1823264: 
2025-07-17T11:07:26.9283041Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9283243Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9283447Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9283641Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9284205Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9284411Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9284619Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9284686Z 
2025-07-17T11:07:26.9284778Z SIGABRT(6), PID: 1822836, Thread 1822931: 
2025-07-17T11:07:26.9285252Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9285457Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9285722Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9285926Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9286413Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9286622Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9286832Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9286838Z 
2025-07-17T11:07:26.9286928Z SIGABRT(6), PID: 1822842, Thread 1823266: 
2025-07-17T11:07:26.9287394Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9287602Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9287803Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9288001Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9288494Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9288700Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9288974Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9288979Z 
2025-07-17T11:07:26.9289072Z SIGABRT(6), PID: 1822836, Thread 1822933: 
2025-07-17T11:07:26.9289539Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9289746Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9289950Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9290150Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9290641Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9290854Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9291060Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9291064Z 
2025-07-17T11:07:26.9291153Z SIGABRT(6), PID: 1822842, Thread 1823268: 
2025-07-17T11:07:26.9291618Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9291890Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9292089Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9292285Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9292781Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9293060Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9293270Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9293274Z 
2025-07-17T11:07:26.9293366Z SIGABRT(6), PID: 1822839, Thread 1827737: 
2025-07-17T11:07:26.9293915Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9294129Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9294333Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9294534Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9294915Z frame #4: <unknown function> + 0x5a02023 (0x7f6c4749f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9295485Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c474a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9296151Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c4749a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9296540Z frame #7: <unknown function> + 0x79d655 (0x7f6c5898a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9296775Z frame #8: <unknown function> + 0xdc253 (0x7f6c0f9d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9296986Z frame #9: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9297286Z frame #10: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9297292Z 
2025-07-17T11:07:26.9297386Z SIGABRT(6), PID: 1822842, Thread 1823270: 
2025-07-17T11:07:26.9297863Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9298072Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9298280Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9298477Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9298981Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9299191Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9299401Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9299406Z 
2025-07-17T11:07:26.9299494Z SIGABRT(6), PID: 1822836, Thread 1822935: 
2025-07-17T11:07:26.9299967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9300172Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9300463Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9300664Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9301162Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9301457Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9301672Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9301676Z 
2025-07-17T11:07:26.9301766Z SIGABRT(6), PID: 1822842, Thread 1823272: 
2025-07-17T11:07:26.9302305Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9302514Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9302720Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9302915Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9303414Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9303621Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9303829Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9303833Z 
2025-07-17T11:07:26.9303920Z SIGABRT(6), PID: 1822836, Thread 1822938: 
2025-07-17T11:07:26.9304392Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9304600Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9304804Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9305000Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9305564Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9305776Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9305988Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9305992Z 
2025-07-17T11:07:26.9306087Z SIGABRT(6), PID: 1822842, Thread 1823274: 
2025-07-17T11:07:26.9306555Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9306763Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9306964Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9307162Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9307654Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9307872Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9308089Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9308093Z 
2025-07-17T11:07:26.9308256Z SIGABRT(6), PID: 1822836, Thread 1822941: 
2025-07-17T11:07:26.9308733Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9308951Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9309156Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9309426Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9309929Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9310140Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9310412Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9310419Z 
2025-07-17T11:07:26.9310517Z SIGABRT(6), PID: 1822842, Thread 1823276: 
2025-07-17T11:07:26.9310990Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9311198Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9311404Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9311607Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9312109Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9312322Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9312533Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9312537Z 
2025-07-17T11:07:26.9312636Z SIGABRT(6), PID: 1822836, Thread 1822944: 
2025-07-17T11:07:26.9313101Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9313309Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9313578Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9313784Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9314277Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9314488Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9314697Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9314701Z 
2025-07-17T11:07:26.9314792Z SIGABRT(6), PID: 1822842, Thread 1823278: 
2025-07-17T11:07:26.9315258Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9315470Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9315677Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9315877Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9316372Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9316652Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9316865Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9316869Z 
2025-07-17T11:07:26.9316960Z SIGABRT(6), PID: 1822836, Thread 1822947: 
2025-07-17T11:07:26.9317433Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9317703Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9317912Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9318112Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9318668Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9318887Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9319097Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9319101Z 
2025-07-17T11:07:26.9319190Z SIGABRT(6), PID: 1822842, Thread 1823280: 
2025-07-17T11:07:26.9319669Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9319874Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9320078Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9320276Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9320768Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9320979Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9321191Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9321197Z 
2025-07-17T11:07:26.9321285Z SIGABRT(6), PID: 1822839, Thread 1827738: 
2025-07-17T11:07:26.9321819Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9322028Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9322235Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9322431Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9322817Z frame #4: <unknown function> + 0x5a02023 (0x7f6c4749f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9323387Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c474a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9323992Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c4749a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9324377Z frame #7: <unknown function> + 0x79d655 (0x7f6c5898a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9324619Z frame #8: <unknown function> + 0xdc253 (0x7f6c0f9d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9324828Z frame #9: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9325111Z frame #10: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9325118Z 
2025-07-17T11:07:26.9325208Z SIGABRT(6), PID: 1822836, Thread 1822950: 
2025-07-17T11:07:26.9325683Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9325980Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9326184Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9326383Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9326954Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9327170Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9327380Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9327384Z 
2025-07-17T11:07:26.9327476Z SIGABRT(6), PID: 1822842, Thread 1823283: 
2025-07-17T11:07:26.9327943Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9328154Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9328358Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9328557Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9329053Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9329266Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9329471Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9329475Z 
2025-07-17T11:07:26.9329566Z SIGABRT(6), PID: 1822842, Thread 1823286: 
2025-07-17T11:07:26.9330110Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9330324Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9330529Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9330729Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9331222Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9331432Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9331639Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9331644Z 
2025-07-17T11:07:26.9331735Z SIGABRT(6), PID: 1822836, Thread 1822953: 
2025-07-17T11:07:26.9332201Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9332413Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9332620Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9332820Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9333393Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9333604Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9333815Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9333883Z 
2025-07-17T11:07:26.9333978Z SIGABRT(6), PID: 1822842, Thread 1823288: 
2025-07-17T11:07:26.9334455Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9334662Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9334923Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9335128Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9335621Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9335830Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9336116Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9336120Z 
2025-07-17T11:07:26.9336215Z SIGABRT(6), PID: 1822836, Thread 1822956: 
2025-07-17T11:07:26.9336685Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9336891Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9337100Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9337297Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9337795Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9338087Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9338302Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9338306Z 
2025-07-17T11:07:26.9338396Z SIGABRT(6), PID: 1822842, Thread 1823290: 
2025-07-17T11:07:26.9338870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9339078Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9339283Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9339479Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9339974Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9340185Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9340396Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9340400Z 
2025-07-17T11:07:26.9340488Z SIGABRT(6), PID: 1822836, Thread 1822958: 
2025-07-17T11:07:26.9340960Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9341267Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9341476Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9341673Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9342172Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9342455Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9342665Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9342668Z 
2025-07-17T11:07:26.9342757Z SIGABRT(6), PID: 1822842, Thread 1823292: 
2025-07-17T11:07:26.9343301Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9343513Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9343719Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9343915Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9344422Z frame #4: <unknown function> + 0x3475ab (0x7f026be325ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9344629Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9344842Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9344846Z 
2025-07-17T11:07:26.9344937Z SIGABRT(6), PID: 1822836, Thread 1822962: 
2025-07-17T11:07:26.9345407Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9345614Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9345819Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9346019Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9346591Z frame #4: <unknown function> + 0x3475ab (0x7fe0f59855ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9346804Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9347020Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9347024Z 
2025-07-17T11:07:26.9347115Z SIGABRT(6), PID: 1822842, Thread 1823892: 
2025-07-17T11:07:26.9347585Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9348040Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f017d69208a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9348250Z frame #2: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9348447Z frame #3: pthread_kill + 0x12c (0x7f026ef9c9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9348614Z frame #4: raise + 0x16 (0x7f026ef48476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9348776Z frame #5: abort + 0xd3 (0x7f026ef2e7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9349016Z frame #6: <unknown function> + 0x972cc (0x7f00eb21b2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9349321Z frame #7: <unknown function> + 0x95f47 (0x7f00eb219f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9349549Z frame #8: <unknown function> + 0x34b71 (0x7f00eb1b8b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9349761Z frame #9: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9349974Z frame #10: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9350051Z 
2025-07-17T11:07:26.9350150Z SIGABRT(6), PID: 1822836, Thread 1823906: 
2025-07-17T11:07:26.9350624Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9351139Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fe0071e608a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9351352Z frame #2: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9351549Z frame #3: pthread_kill + 0x12c (0x7fe0f8aef9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9351710Z frame #4: raise + 0x16 (0x7fe0f8a9b476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9351869Z frame #5: abort + 0xd3 (0x7fe0f8a817f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9352102Z frame #6: <unknown function> + 0x972cc (0x7fdf74d6f2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9352336Z frame #7: <unknown function> + 0x95f47 (0x7fdf74d6df47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9352561Z frame #8: <unknown function> + 0x34b71 (0x7fdf74d0cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9352774Z frame #9: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9352989Z frame #10: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9352994Z 
2025-07-17T11:07:26.9353089Z SIGABRT(6), PID: 1822842, Thread 1826218: 
2025-07-17T11:07:26.9353556Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9353770Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9353933Z frame #2: __poll + 0x4f (0x7f026f01ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9354423Z frame #3: <unknown function> + 0x489f (0x7f026eb2a89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.9354655Z frame #4: <unknown function> + 0x1483bc (0x562d2273b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9354903Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9355130Z frame #6: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9355369Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9355589Z frame #8: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9355835Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x562d22723a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9356065Z frame #10: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9356315Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x562d22721c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9356540Z frame #12: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9356776Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9357000Z frame #14: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9357329Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9357547Z frame #16: <unknown function> + 0x147734 (0x562d2273a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9357768Z frame #17: <unknown function> + 0x21ca82 (0x562d2280fa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9357988Z frame #18: <unknown function> + 0x21c9e4 (0x562d2280f9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9358272Z frame #19: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9358483Z frame #20: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9358487Z 
2025-07-17T11:07:26.9358582Z SIGABRT(6), PID: 1822839, Thread 1827739: 
2025-07-17T11:07:26.9359131Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9359351Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9359558Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9359765Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9360149Z frame #4: <unknown function> + 0x5a02023 (0x7f6c4749f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9360730Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c474a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9361327Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c4749a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9361717Z frame #7: <unknown function> + 0x79d655 (0x7f6c5898a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9361948Z frame #8: <unknown function> + 0xdc253 (0x7f6c0f9d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9362159Z frame #9: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9362371Z frame #10: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9362441Z 
2025-07-17T11:07:26.9362540Z SIGABRT(6), PID: 1822836, Thread 1827213: 
2025-07-17T11:07:26.9363013Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9363225Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9363391Z frame #2: __poll + 0x4f (0x7fe0f8b71bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9363798Z frame #3: <unknown function> + 0x489f (0x7fe0f867d89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.9364023Z frame #4: <unknown function> + 0x1483bc (0x56276fa2e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9364266Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9364495Z frame #6: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9364731Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9364953Z frame #8: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9365197Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x56276fa16a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9365424Z frame #10: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9365738Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x56276fa14c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9365965Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9366201Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9366488Z frame #14: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9366727Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9366946Z frame #16: <unknown function> + 0x147734 (0x56276fa2d734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9367226Z frame #17: <unknown function> + 0x21ca82 (0x56276fb02a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9367447Z frame #18: <unknown function> + 0x21c9e4 (0x56276fb029e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9367668Z frame #19: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9367879Z frame #20: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9367883Z 
2025-07-17T11:07:26.9367980Z SIGABRT(6), PID: 1822842, Thread 1827625: 
2025-07-17T11:07:26.9368457Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9368671Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9368876Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9369101Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f026ef9a2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9369543Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f01a93da048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9369973Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f01a93dcbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9370205Z frame #6: <unknown function> + 0xdc253 (0x7f017b8fe253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9370479Z frame #7: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9370698Z frame #8: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9370702Z 
2025-07-17T11:07:26.9370794Z SIGABRT(6), PID: 1822842, Thread 1827626: 
2025-07-17T11:07:26.9371268Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9371475Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9371680Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9371902Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f026ef9a2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9372375Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f01a93d4741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9372607Z frame #5: <unknown function> + 0xdc253 (0x7f017b8fe253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9372816Z frame #6: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9373024Z frame #7: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9373028Z 
2025-07-17T11:07:26.9373122Z SIGABRT(6), PID: 1822842, Thread 1827627: 
2025-07-17T11:07:26.9373587Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9373868Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9374035Z frame #2: ioctl + 0x3f (0x7f026f02094f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9374273Z frame #3: <unknown function> + 0x13eef0 (0x7f00eb2c2ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9374576Z frame #4: <unknown function> + 0x1378c4 (0x7f00eb2bb8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9374811Z frame #5: <unknown function> + 0x95b2b (0x7f00eb219b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9375039Z frame #6: <unknown function> + 0x34b71 (0x7f00eb1b8b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9375308Z frame #7: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9375521Z frame #8: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9375527Z 
2025-07-17T11:07:26.9375622Z SIGABRT(6), PID: 1822839, Thread 1827740: 
2025-07-17T11:07:26.9376169Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9376380Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9376584Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9376789Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9377162Z frame #4: <unknown function> + 0x5a02023 (0x7f6c4749f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9377732Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c474a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9378327Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c4749a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9378711Z frame #7: <unknown function> + 0x79d655 (0x7f6c5898a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9379137Z frame #8: <unknown function> + 0xdc253 (0x7f6c0f9d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9379354Z frame #9: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9379568Z frame #10: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9379572Z 
2025-07-17T11:07:26.9379666Z SIGABRT(6), PID: 1822836, Thread 1827214: 
2025-07-17T11:07:26.9380134Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9380345Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9380528Z frame #2: epoll_wait + 0x5e (0x7fe0f8b7ee2e in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9380861Z frame #3: uv__io_poll + 0x6bd (0x7fe04973c44d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9381181Z frame #4: uv_run + 0x12a (0x7fe04973036a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9381558Z frame #5: <unknown function> + 0x658a08a (0x7fe03daa408a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9381789Z frame #6: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9382079Z frame #7: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9382294Z frame #8: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9382298Z 
2025-07-17T11:07:26.9382388Z SIGABRT(6), PID: 1822842, Thread 1827647: 
2025-07-17T11:07:26.9382859Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9383150Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9383324Z frame #2: __poll + 0x4f (0x7f026f01ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9383536Z frame #3: <unknown function> + 0xd4540 (0x7f0133432540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9383812Z frame #4: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9384022Z frame #5: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9384028Z 
2025-07-17T11:07:26.9384121Z SIGABRT(6), PID: 1822842, Thread 1827656: 
2025-07-17T11:07:26.9384588Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9384796Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9384962Z frame #2: __poll + 0x4f (0x7f026f01ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9385178Z frame #3: <unknown function> + 0xd65a2 (0x7f01334345a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9385384Z frame #4: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9385593Z frame #5: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9385597Z 
2025-07-17T11:07:26.9385687Z SIGABRT(6), PID: 1822842, Thread 1827666: 
2025-07-17T11:07:26.9386154Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9386359Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9386560Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9386850Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9387067Z frame #4: <unknown function> + 0xd1e7f (0x7f013342fe7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9387272Z frame #5: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9387480Z frame #6: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9387486Z 
2025-07-17T11:07:26.9387574Z SIGABRT(6), PID: 1822836, Thread 1827633: 
2025-07-17T11:07:26.9388045Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9388251Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9388457Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9388689Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fe0f8aed2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9389133Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fe032f2e048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9389558Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fe032f30bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9389789Z frame #6: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9390068Z frame #7: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9390279Z frame #8: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9390283Z 
2025-07-17T11:07:26.9390371Z SIGABRT(6), PID: 1822842, Thread 1827674: 
2025-07-17T11:07:26.9390904Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9391111Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9391318Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9391589Z frame #3: <unknown function> + 0x9ca53 (0x7f026efa2a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9391850Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x562d22707bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9392080Z frame #5: <unknown function> + 0x1d314f (0x562d227c614f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9392302Z frame #6: <unknown function> + 0x1d2f6b (0x562d227c5f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9392521Z frame #7: <unknown function> + 0x6df36 (0x562d22660f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9392769Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9392994Z frame #9: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9393237Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9393463Z frame #11: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9393699Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9393925Z frame #13: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9394162Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9394385Z frame #15: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9394686Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9394908Z frame #17: <unknown function> + 0x147734 (0x562d2273a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9395130Z frame #18: <unknown function> + 0x21ca82 (0x562d2280fa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9395349Z frame #19: <unknown function> + 0x21c9e4 (0x562d2280f9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9395564Z frame #20: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9395777Z frame #21: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9395782Z 
2025-07-17T11:07:26.9395877Z SIGABRT(6), PID: 1822842, Thread 1827687: 
2025-07-17T11:07:26.9396345Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9396560Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9396723Z frame #2: read + 0x4c (0x7f026f01a81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9396905Z frame #3: _Py_read + 0x4c (0x562d227cda6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9397129Z frame #4: <unknown function> + 0x208c3b (0x562d227fbc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9397349Z frame #5: <unknown function> + 0x14648d (0x562d2273948d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9397658Z frame #6: PyObject_VectorcallMethod + 0x85 (0x562d22748215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9397879Z frame #7: <unknown function> + 0x208ae7 (0x562d227fbae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9398093Z frame #8: <unknown function> + 0x208472 (0x562d227fb472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9398313Z frame #9: <unknown function> + 0x1da6ee (0x562d227cd6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9398598Z frame #10: <unknown function> + 0x1483bc (0x562d2273b3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9398841Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9399062Z frame #12: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9399360Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x562d2271f3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9399582Z frame #14: <unknown function> + 0x147734 (0x562d2273a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9399830Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x562d22721c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9400051Z frame #16: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9400285Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9400512Z frame #18: _PyFunction_Vectorcall + 0x6c (0x562d2272f0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9400746Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x562d2271f7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9400962Z frame #20: <unknown function> + 0x147734 (0x562d2273a734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9401184Z frame #21: <unknown function> + 0x21ca82 (0x562d2280fa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9401403Z frame #22: <unknown function> + 0x21c9e4 (0x562d2280f9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9401616Z frame #23: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9401824Z frame #24: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9401828Z 
2025-07-17T11:07:26.9401922Z SIGABRT(6), PID: 1822836, Thread 1827634: 
2025-07-17T11:07:26.9402460Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9402679Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9402886Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9403115Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fe0f8aed2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9403588Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fe032f28741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9403817Z frame #5: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9404024Z frame #6: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9404237Z frame #7: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9404241Z 
2025-07-17T11:07:26.9404333Z SIGABRT(6), PID: 1822836, Thread 1827636: 
2025-07-17T11:07:26.9404802Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9405011Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9405175Z frame #2: ioctl + 0x3f (0x7fe0f8b7394f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9405493Z frame #3: <unknown function> + 0x13eef0 (0x7fdf74e16ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9405726Z frame #4: <unknown function> + 0x1378c4 (0x7fdf74e0f8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9405959Z frame #5: <unknown function> + 0x95b2b (0x7fdf74d6db2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9406248Z frame #6: <unknown function> + 0x34b71 (0x7fdf74d0cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9406460Z frame #7: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9406667Z frame #8: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9406671Z 
2025-07-17T11:07:26.9406763Z SIGABRT(6), PID: 1822839, Thread 1827741: 
2025-07-17T11:07:26.9407284Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f6c11768b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9407499Z frame #1: <unknown function> + 0x42520 (0x7f6d0301f520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9407705Z frame #2: <unknown function> + 0x91117 (0x7f6d0306e117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9407907Z frame #3: pthread_cond_wait + 0x211 (0x7f6d03070a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9408290Z frame #4: <unknown function> + 0x5a02023 (0x7f6c4749f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9408859Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f6c474a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9409461Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f6c4749a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9409847Z frame #7: <unknown function> + 0x79d655 (0x7f6c5898a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9410075Z frame #8: <unknown function> + 0xdc253 (0x7f6c0f9d5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9410285Z frame #9: <unknown function> + 0x94ac3 (0x7f6d03071ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9410562Z frame #10: <unknown function> + 0x126850 (0x7f6d03103850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9410569Z 
2025-07-17T11:07:26.9410664Z SIGABRT(6), PID: 1822836, Thread 1827652: 
2025-07-17T11:07:26.9411135Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9411346Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9411509Z frame #2: __poll + 0x4f (0x7fe0f8b71bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9411725Z frame #3: <unknown function> + 0xd4540 (0x7fdfbcf86540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9411932Z frame #4: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9412142Z frame #5: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9412147Z 
2025-07-17T11:07:26.9412236Z SIGABRT(6), PID: 1822836, Thread 1827661: 
2025-07-17T11:07:26.9412708Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9412913Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9413079Z frame #2: __poll + 0x4f (0x7fe0f8b71bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9413293Z frame #3: <unknown function> + 0xd65a2 (0x7fdfbcf885a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9413573Z frame #4: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9413786Z frame #5: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9413791Z 
2025-07-17T11:07:26.9413886Z SIGABRT(6), PID: 1822836, Thread 1827669: 
2025-07-17T11:07:26.9414358Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9414636Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9414843Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9415130Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9415354Z frame #4: <unknown function> + 0xd1e7f (0x7fdfbcf83e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9415567Z frame #5: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9415777Z frame #6: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9415781Z 
2025-07-17T11:07:26.9415873Z SIGABRT(6), PID: 1822836, Thread 1827672: 
2025-07-17T11:07:26.9416414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9416628Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9416836Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9417048Z frame #3: <unknown function> + 0x9ca53 (0x7fe0f8af5a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9417305Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x56276f9fabc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9417534Z frame #5: <unknown function> + 0x1d314f (0x56276fab914f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9417755Z frame #6: <unknown function> + 0x1d2f6b (0x56276fab8f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9417977Z frame #7: <unknown function> + 0x6df36 (0x56276f953f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9418302Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9418533Z frame #9: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9418772Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9419001Z frame #11: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9419236Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9419468Z frame #13: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9419699Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9419925Z frame #15: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9420159Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9420382Z frame #17: <unknown function> + 0x147734 (0x56276fa2d734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9420599Z frame #18: <unknown function> + 0x21ca82 (0x56276fb02a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9420820Z frame #19: <unknown function> + 0x21c9e4 (0x56276fb029e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9421036Z frame #20: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9421334Z frame #21: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9421338Z 
2025-07-17T11:07:26.9421431Z SIGABRT(6), PID: 1822836, Thread 1827685: 
2025-07-17T11:07:26.9421906Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9422201Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9422368Z frame #2: read + 0x4c (0x7fe0f8b6d81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9422547Z frame #3: _Py_read + 0x4c (0x56276fac0a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9422769Z frame #4: <unknown function> + 0x208c3b (0x56276faeec3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9423077Z frame #5: <unknown function> + 0x14648d (0x56276fa2c48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9423328Z frame #6: PyObject_VectorcallMethod + 0x85 (0x56276fa3b215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9423553Z frame #7: <unknown function> + 0x208ae7 (0x56276faeeae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9423769Z frame #8: <unknown function> + 0x208472 (0x56276faee472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9423991Z frame #9: <unknown function> + 0x1da6ee (0x56276fac06ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9424214Z frame #10: <unknown function> + 0x1483bc (0x56276fa2e3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9424455Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9424680Z frame #12: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9424917Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x56276fa123bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9425134Z frame #14: <unknown function> + 0x147734 (0x56276fa2d734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9425384Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x56276fa14c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9425612Z frame #16: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9425853Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9426144Z frame #18: _PyFunction_Vectorcall + 0x6c (0x56276fa220ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9426382Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x56276fa127b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9426605Z frame #20: <unknown function> + 0x147734 (0x56276fa2d734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9426826Z frame #21: <unknown function> + 0x21ca82 (0x56276fb02a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9427048Z frame #22: <unknown function> + 0x21c9e4 (0x56276fb029e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9427261Z frame #23: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9427477Z frame #24: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9427481Z 
2025-07-17T11:07:26.9427580Z SIGABRT(6), PID: 1822842, Thread 1827726: 
2025-07-17T11:07:26.9428052Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9428266Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9428471Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9428673Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9429049Z frame #4: <unknown function> + 0x5a02023 (0x7f01b33c8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9429699Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f01b33cc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9430303Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f01b33c31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9430754Z frame #7: <unknown function> + 0x79d655 (0x7f01c48af655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9430989Z frame #8: <unknown function> + 0xdc253 (0x7f017b8fe253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9431262Z frame #9: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9431490Z frame #10: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9431494Z 
2025-07-17T11:07:26.9431589Z SIGABRT(6), PID: 1822842, Thread 1827727: 
2025-07-17T11:07:26.9432065Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9432278Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9432486Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9432691Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9433062Z frame #4: <unknown function> + 0x5a02023 (0x7f01b33c8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9433625Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f01b33cc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9434223Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f01b33c31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9434672Z frame #7: <unknown function> + 0x79d655 (0x7f01c48af655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9434910Z frame #8: <unknown function> + 0xdc253 (0x7f017b8fe253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9435127Z frame #9: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9435341Z frame #10: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9435347Z 
2025-07-17T11:07:26.9435441Z SIGABRT(6), PID: 1822836, Thread 1827707: 
2025-07-17T11:07:26.9435917Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9436129Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9436335Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9436540Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9436910Z frame #4: <unknown function> + 0x5a02023 (0x7fe03cf1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9437473Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe03cf201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9438066Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe03cf171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9438518Z frame #7: <unknown function> + 0x79d655 (0x7fe04e407655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9438748Z frame #8: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9439026Z frame #9: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9439245Z frame #10: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9439249Z 
2025-07-17T11:07:26.9439341Z SIGABRT(6), PID: 1822842, Thread 1827728: 
2025-07-17T11:07:26.9439889Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9440099Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9440307Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9440507Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9440876Z frame #4: <unknown function> + 0x5a02023 (0x7f01b33c8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9441439Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f01b33cc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9442031Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f01b33c31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9442412Z frame #7: <unknown function> + 0x79d655 (0x7f01c48af655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9442646Z frame #8: <unknown function> + 0xdc253 (0x7f017b8fe253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9442853Z frame #9: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9443067Z frame #10: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9443134Z 
2025-07-17T11:07:26.9443231Z SIGABRT(6), PID: 1822836, Thread 1827709: 
2025-07-17T11:07:26.9443705Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9443913Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9444121Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9444320Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9444692Z frame #4: <unknown function> + 0x5a02023 (0x7fe03cf1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9445250Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe03cf201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9445845Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe03cf171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9446225Z frame #7: <unknown function> + 0x79d655 (0x7fe04e407655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9446451Z frame #8: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9446729Z frame #9: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9446942Z frame #10: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9446946Z 
2025-07-17T11:07:26.9447036Z SIGABRT(6), PID: 1822836, Thread 1827711: 
2025-07-17T11:07:26.9447594Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9447807Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9448009Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9448268Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9448642Z frame #4: <unknown function> + 0x5a02023 (0x7fe03cf1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9449206Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe03cf201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9449799Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe03cf171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9450179Z frame #7: <unknown function> + 0x79d655 (0x7fe04e407655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9450405Z frame #8: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9450619Z frame #9: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9450834Z frame #10: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9450838Z 
2025-07-17T11:07:26.9450929Z SIGABRT(6), PID: 1822842, Thread 1827729: 
2025-07-17T11:07:26.9451399Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9451608Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9451878Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9452083Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9452452Z frame #4: <unknown function> + 0x5a02023 (0x7f01b33c8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9453014Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f01b33cc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9453607Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f01b33c31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9453989Z frame #7: <unknown function> + 0x79d655 (0x7f01c48af655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9454220Z frame #8: <unknown function> + 0xdc253 (0x7f017b8fe253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9454431Z frame #9: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9454641Z frame #10: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9454650Z 
2025-07-17T11:07:26.9454741Z SIGABRT(6), PID: 1822836, Thread 1827713: 
2025-07-17T11:07:26.9455286Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9455496Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9455705Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9456036Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9456411Z frame #4: <unknown function> + 0x5a02023 (0x7fe03cf1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9457046Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe03cf201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9457645Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe03cf171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9458023Z frame #7: <unknown function> + 0x79d655 (0x7fe04e407655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9458252Z frame #8: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9458467Z frame #9: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9458681Z frame #10: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9458685Z 
2025-07-17T11:07:26.9458775Z SIGABRT(6), PID: 1822842, Thread 1827730: 
2025-07-17T11:07:26.9459245Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9459451Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9459654Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9459848Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9460217Z frame #4: <unknown function> + 0x5a02023 (0x7f01b33c8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9460853Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f01b33cc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9461450Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f01b33c31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9461830Z frame #7: <unknown function> + 0x79d655 (0x7f01c48af655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9462062Z frame #8: <unknown function> + 0xdc253 (0x7f017b8fe253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9462266Z frame #9: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9462481Z frame #10: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9462488Z 
2025-07-17T11:07:26.9462578Z SIGABRT(6), PID: 1822836, Thread 1827714: 
2025-07-17T11:07:26.9463049Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9463254Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9463463Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9463742Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9464122Z frame #4: <unknown function> + 0x5a02023 (0x7fe03cf1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9464684Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe03cf201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9465350Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe03cf171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9465792Z frame #7: <unknown function> + 0x79d655 (0x7fe04e407655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9466024Z frame #8: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9466235Z frame #9: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9466446Z frame #10: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9466450Z 
2025-07-17T11:07:26.9466542Z SIGABRT(6), PID: 1822842, Thread 1827731: 
2025-07-17T11:07:26.9467017Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9467226Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9467430Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9467631Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9467998Z frame #4: <unknown function> + 0x5a02023 (0x7f01b33c8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9468563Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f01b33cc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9469226Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f01b33c31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9469614Z frame #7: <unknown function> + 0x79d655 (0x7f01c48af655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9469843Z frame #8: <unknown function> + 0xdc253 (0x7f017b8fe253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9470053Z frame #9: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9470265Z frame #10: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9470269Z 
2025-07-17T11:07:26.9470360Z SIGABRT(6), PID: 1822836, Thread 1827715: 
2025-07-17T11:07:26.9470825Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9471040Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9471248Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9471451Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9471820Z frame #4: <unknown function> + 0x5a02023 (0x7fe03cf1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9472384Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe03cf201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9473048Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe03cf171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9473432Z frame #7: <unknown function> + 0x79d655 (0x7fe04e407655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9473732Z frame #8: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9473940Z frame #9: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9474156Z frame #10: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9474218Z 
2025-07-17T11:07:26.9474311Z SIGABRT(6), PID: 1822842, Thread 1827732: 
2025-07-17T11:07:26.9474785Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9474991Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9475196Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9475396Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9475766Z frame #4: <unknown function> + 0x5a02023 (0x7f01b33c8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9476321Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f01b33cc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9476916Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f01b33c31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9477296Z frame #7: <unknown function> + 0x79d655 (0x7f01c48af655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9477526Z frame #8: <unknown function> + 0xdc253 (0x7f017b8fe253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9477799Z frame #9: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9478017Z frame #10: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9478021Z 
2025-07-17T11:07:26.9478110Z SIGABRT(6), PID: 1822836, Thread 1827716: 
2025-07-17T11:07:26.9478584Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9478791Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9478996Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9479193Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9479565Z frame #4: <unknown function> + 0x5a02023 (0x7fe03cf1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9480132Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe03cf201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9480729Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe03cf171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9481110Z frame #7: <unknown function> + 0x79d655 (0x7fe04e407655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9481429Z frame #8: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9481637Z frame #9: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9481854Z frame #10: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9481917Z 
2025-07-17T11:07:26.9482011Z SIGABRT(6), PID: 1822842, Thread 1827733: 
2025-07-17T11:07:26.9482479Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f017d691b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9482689Z frame #1: <unknown function> + 0x42520 (0x7f026ef48520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9482955Z frame #2: <unknown function> + 0x91117 (0x7f026ef97117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9483166Z frame #3: pthread_cond_wait + 0x211 (0x7f026ef99a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9483536Z frame #4: <unknown function> + 0x5a02023 (0x7f01b33c8023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9484097Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f01b33cc1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9484689Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f01b33c31d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9485072Z frame #7: <unknown function> + 0x79d655 (0x7f01c48af655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9485298Z frame #8: <unknown function> + 0xdc253 (0x7f017b8fe253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9485508Z frame #9: <unknown function> + 0x94ac3 (0x7f026ef9aac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9485717Z frame #10: <unknown function> + 0x126850 (0x7f026f02c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9485722Z 
2025-07-17T11:07:26.9485815Z SIGABRT(6), PID: 1822836, Thread 1827717: 
2025-07-17T11:07:26.9486345Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fe0071e5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9486559Z frame #1: <unknown function> + 0x42520 (0x7fe0f8a9b520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9486764Z frame #2: <unknown function> + 0x91117 (0x7fe0f8aea117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9486967Z frame #3: pthread_cond_wait + 0x211 (0x7fe0f8aeca41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9487336Z frame #4: <unknown function> + 0x5a02023 (0x7fe03cf1c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9487899Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fe03cf201ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9488488Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fe03cf171d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:26.9488876Z frame #7: <unknown function> + 0x79d655 (0x7fe04e407655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9489104Z frame #8: <unknown function> + 0xdc253 (0x7fe005452253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9489318Z frame #9: <unknown function> + 0x94ac3 (0x7fe0f8aedac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9489530Z frame #10: <unknown function> + 0x126850 (0x7fe0f8b7f850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9489599Z 
2025-07-17T11:07:26.9489707Z ('RERUN', {'yellow': True}) [25.5458s] [100%]
2025-07-17T11:07:26.9490357Z distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph I0717 11:03:25.956000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1828342
2025-07-17T11:07:26.9490783Z I0717 11:03:25.957000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1828343
2025-07-17T11:07:26.9491132Z I0717 11:03:25.958000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1828344
2025-07-17T11:07:26.9491476Z I0717 11:03:26.002000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1828345
2025-07-17T11:07:26.9491875Z I0717 11:03:26.003000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1828346
2025-07-17T11:07:26.9492226Z I0717 11:03:26.004000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1828347
2025-07-17T11:07:26.9492571Z I0717 11:03:26.006000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1828348
2025-07-17T11:07:26.9492912Z I0717 11:03:26.007000 1817336 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1828349
2025-07-17T11:07:26.9493928Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.9494012Z   import pkg_resources
2025-07-17T11:07:26.9494989Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.9495070Z   import pkg_resources
2025-07-17T11:07:26.9496210Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.9496299Z   import pkg_resources
2025-07-17T11:07:26.9497285Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.9497363Z   import pkg_resources
2025-07-17T11:07:26.9498345Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.9498423Z   import pkg_resources
2025-07-17T11:07:26.9499403Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.9499485Z   import pkg_resources
2025-07-17T11:07:26.9500465Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.9500625Z   import pkg_resources
2025-07-17T11:07:26.9501599Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:26.9501752Z   import pkg_resources
2025-07-17T11:07:26.9502237Z [rank6]:W0717 11:03:47.445000 1828348 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.9502778Z [rank2]:W0717 11:03:47.445000 1828344 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.9503251Z [rank1]:W0717 11:03:47.445000 1828343 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.9503715Z [rank0]:W0717 11:03:47.445000 1828342 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.9504183Z [rank4]:W0717 11:03:47.446000 1828346 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.9504645Z [rank3]:W0717 11:03:47.447000 1828345 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.9505114Z [rank5]:W0717 11:03:47.449000 1828347 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.9505580Z [rank7]:W0717 11:03:47.450000 1828349 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:26.9505867Z Memory access fault by GPU node-3 (Agent handle: 0x559fd4f7a960) on address 0x7f8742a03000. Reason: Unknown.
2025-07-17T11:07:26.9506207Z Memory access fault by GPU node-5 (Agent handle: 0x5638624343a0) on address 0x7f565fe1d000. Reason: Unknown.
2025-07-17T11:07:26.9506480Z Memory access fault by GPU node-2 (Agent handle: 0x559d8fc88480) on address 0x7fe41200d000. Reason: Unknown.
2025-07-17T11:07:26.9506751Z Memory access fault by GPU node-4 (Agent handle: 0x555d031cfa40) on address 0x7f9583e04000. Reason: Unknown.
2025-07-17T11:07:26.9507019Z Memory access fault by GPU node-7 (Agent handle: 0x55f482ba30d0) on address 0x7f1117e06000. Reason: Unknown.
2025-07-17T11:07:26.9507278Z Memory access fault by GPU node-6 (Agent handle: 0x556882592220) on address 0x7f3983e23000. Reason: Unknown.
2025-07-17T11:07:26.9507544Z Memory access fault by GPU node-8 (Agent handle: 0x55f394bf5e40) on address 0x7f96d5c1f000. Reason: Unknown.
2025-07-17T11:07:26.9507807Z Memory access fault by GPU node-9 (Agent handle: 0x55d83dcbc890) on address 0x7f8377407000. Reason: Unknown.
2025-07-17T11:07:26.9507905Z SIGABRT(6), PID: 1828343, Thread 1828343: 
2025-07-17T11:07:26.9508444Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9508675Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9508914Z frame #2: <unknown function> + 0x773e7 (0x7f8f7e6093e7 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9509147Z frame #3: <unknown function> + 0x76c8e (0x7f8f7e608c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9509450Z frame #4: <unknown function> + 0x6ad69 (0x7f8f7e5fcd69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9509671Z frame #5: <unknown function> + 0x3f54db (0x7f900f2614db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.9509885Z frame #6: <unknown function> + 0x3e11d8 (0x7f900f24d1d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.9510095Z frame #7: <unknown function> + 0xb2f86 (0x7f900ef1ef86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.9510371Z frame #8: <unknown function> + 0x8e501 (0x7f900eefa501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.9510760Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f9039d017ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.9511329Z frame #10: <unknown function> + 0xd307d9 (0x7f90582547d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9511568Z frame #11: <unknown function> + 0x139902 (0x559fcb4b0902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9511823Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x559fcb4a7a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9512056Z frame #13: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9512297Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x559fcb4a7a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9512529Z frame #15: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9512771Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9512995Z frame #17: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9513230Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9513456Z frame #19: <unknown function> + 0x1474b2 (0x559fcb4be4b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9513660Z frame #20: PyObject_Call + 0xbc (0x559fcb4bee4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9513883Z frame #21: <unknown function> + 0x1ff69d (0x559fcb57669d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9514114Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x559fcb4ac023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9514355Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x559fcb4a7cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9514670Z frame #24: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9514909Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x559fcb4a33bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9515135Z frame #26: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9515371Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x559fcb4a33bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9515604Z frame #28: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9515845Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x559fcb4a7a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9516105Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x559fcb4ab4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9516330Z frame #31: _PyObject_Call_Prepend + 0x69 (0x559fcb4bcab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9516555Z frame #32: <unknown function> + 0x205949 (0x559fcb57c949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9516780Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x559fcb4ac023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9517022Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x559fcb4a7cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9517244Z frame #35: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9517550Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x559fcb4a33bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9517772Z frame #37: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9518013Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x559fcb4a33bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9518244Z frame #39: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9518545Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x559fcb4a33bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9518770Z frame #41: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9519009Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x559fcb4a43f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9519296Z frame #43: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9519694Z frame #44: <unknown function> + 0x7ab3f7 (0x7f9057ccf3f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9519913Z frame #45: <unknown function> + 0x13bc88 (0x559fcb4b2c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9520124Z frame #46: PyObject_Call + 0x20f (0x559fcb4bef9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9520365Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x559fcb4a898d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9520593Z frame #48: <unknown function> + 0x147608 (0x559fcb4be608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9520833Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x559fcb4a5c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9521057Z frame #50: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9521299Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x559fcb4a33bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9521527Z frame #52: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9521764Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x559fcb4a43f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9521992Z frame #54: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9522228Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x559fcb4a33bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9522520Z frame #56: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9522765Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x559fcb4a5c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9522996Z frame #58: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9523237Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x559fcb4a5c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9523460Z frame #60: <unknown function> + 0x147608 (0x559fcb4be608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9523699Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x559fcb4a5c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9523917Z frame #62: <unknown function> + 0x147608 (0x559fcb4be608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9523921Z 
2025-07-17T11:07:26.9524016Z SIGABRT(6), PID: 1828343, Thread 1828375: 
2025-07-17T11:07:26.9524501Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9524717Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9524925Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9525129Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9525708Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9525921Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9526129Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9526195Z 
2025-07-17T11:07:26.9526293Z SIGABRT(6), PID: 1828343, Thread 1828378: 
2025-07-17T11:07:26.9526768Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9526977Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9527239Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9527444Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9527946Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9528156Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9528368Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9528375Z 
2025-07-17T11:07:26.9528468Z SIGABRT(6), PID: 1828343, Thread 1828381: 
2025-07-17T11:07:26.9528939Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9529147Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9529353Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9529551Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9530045Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9530253Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9530524Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9530529Z 
2025-07-17T11:07:26.9530625Z SIGABRT(6), PID: 1828343, Thread 1828383: 
2025-07-17T11:07:26.9531093Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9531302Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9531505Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9531710Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9532201Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9532415Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9532622Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9532626Z 
2025-07-17T11:07:26.9532719Z SIGABRT(6), PID: 1828343, Thread 1828385: 
2025-07-17T11:07:26.9533195Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9533404Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9533675Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9533872Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9534367Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9534650Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9534857Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9534864Z 
2025-07-17T11:07:26.9534954Z SIGABRT(6), PID: 1828343, Thread 1828388: 
2025-07-17T11:07:26.9535490Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9535698Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9535962Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9536165Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9536664Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9536873Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9537083Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9537087Z 
2025-07-17T11:07:26.9537177Z SIGABRT(6), PID: 1828343, Thread 1828389: 
2025-07-17T11:07:26.9537651Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9537854Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9538056Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9538250Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9538833Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9539045Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9539255Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9539261Z 
2025-07-17T11:07:26.9539350Z SIGABRT(6), PID: 1828343, Thread 1828392: 
2025-07-17T11:07:26.9539826Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9540030Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9540233Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9540429Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9540922Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9541127Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9541337Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9541421Z 
2025-07-17T11:07:26.9541511Z SIGABRT(6), PID: 1828343, Thread 1828394: 
2025-07-17T11:07:26.9541982Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9542186Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9542388Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9542659Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9543154Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9543457Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9543672Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9543679Z 
2025-07-17T11:07:26.9543770Z SIGABRT(6), PID: 1828343, Thread 1828396: 
2025-07-17T11:07:26.9544242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9544445Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9544651Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9544846Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9545341Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9545549Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9545759Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9545764Z 
2025-07-17T11:07:26.9545855Z SIGABRT(6), PID: 1828343, Thread 1828398: 
2025-07-17T11:07:26.9546326Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9546602Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9546807Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9547009Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9547502Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9547712Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9547919Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9547923Z 
2025-07-17T11:07:26.9548017Z SIGABRT(6), PID: 1828343, Thread 1828400: 
2025-07-17T11:07:26.9548487Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9548697Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9548896Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9549090Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9549581Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9549861Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9550070Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9550074Z 
2025-07-17T11:07:26.9550168Z SIGABRT(6), PID: 1828343, Thread 1828402: 
2025-07-17T11:07:26.9550634Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9550907Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9551109Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9551306Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9551858Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9552074Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9552281Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9552285Z 
2025-07-17T11:07:26.9552377Z SIGABRT(6), PID: 1828343, Thread 1828403: 
2025-07-17T11:07:26.9552850Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9553058Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9553259Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9553458Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9553951Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9554157Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9554359Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9554457Z SIGABRT(6), PID: 1828345, Thread 1828345: 
2025-07-17T11:07:26.9554990Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9555202Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9555439Z frame #2: <unknown function> + 0x7741f (0x7f5ea40bf41f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9555676Z frame #3: <unknown function> + 0x76c8e (0x7f5ea40bec8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9555905Z frame #4: <unknown function> + 0x6ad69 (0x7f5ea40b2d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9556124Z frame #5: <unknown function> + 0x3f54db (0x7f5f34d174db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.9556334Z frame #6: <unknown function> + 0x3e11d8 (0x7f5f34d031d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.9556548Z frame #7: <unknown function> + 0xb2f86 (0x7f5f349d4f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.9556756Z frame #8: <unknown function> + 0x8e501 (0x7f5f349b0501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:26.9557145Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f5f5f7b77ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:26.9557537Z frame #10: <unknown function> + 0xd307d9 (0x7f5f7dd0a7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9557845Z frame #11: <unknown function> + 0x139902 (0x563858c77902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9558094Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x563858c6ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9558328Z frame #13: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9558566Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x563858c6ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9558859Z frame #15: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9559096Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9559323Z frame #17: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9559617Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9559846Z frame #19: <unknown function> + 0x1474b2 (0x563858c854b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9560052Z frame #20: PyObject_Call + 0xbc (0x563858c85e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9560273Z frame #21: <unknown function> + 0x1ff69d (0x563858d3d69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9560497Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x563858c73023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9560746Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x563858c6ecc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9560967Z frame #24: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9561209Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x563858c6a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9561432Z frame #26: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9561667Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x563858c6a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9561891Z frame #28: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9562128Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x563858c6ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9562382Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x563858c724b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9562667Z frame #31: _PyObject_Call_Prepend + 0x69 (0x563858c83ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9562886Z frame #32: <unknown function> + 0x205949 (0x563858d43949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9563111Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x563858c73023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9563352Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x563858c6ecc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9563579Z frame #35: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9563815Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x563858c6a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9564041Z frame #37: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9564275Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x563858c6a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9564503Z frame #39: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9564736Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x563858c6a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9564960Z frame #41: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9565201Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x563858c6b3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9565423Z frame #43: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9565874Z frame #44: <unknown function> + 0x7ab3f7 (0x7f5f7d7853f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:26.9566096Z frame #45: <unknown function> + 0x13bc88 (0x563858c79c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9566297Z frame #46: PyObject_Call + 0x20f (0x563858c85f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9566608Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x563858c6f98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9566824Z frame #48: <unknown function> + 0x147608 (0x563858c85608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9567065Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x563858c6cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9567345Z frame #50: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9567589Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x563858c6a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9567815Z frame #52: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9568060Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x563858c6b3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9568285Z frame #54: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9568529Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x563858c6a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9568751Z frame #56: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9568991Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x563858c6cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9569214Z frame #58: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9569454Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x563858c6cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9569670Z frame #60: <unknown function> + 0x147608 (0x563858c85608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9569908Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x563858c6cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9570123Z frame #62: <unknown function> + 0x147608 (0x563858c85608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9570129Z 
2025-07-17T11:07:26.9570133Z 
2025-07-17T11:07:26.9570293Z SIGABRT(6), PID: 1828343, Thread 1828405: 
2025-07-17T11:07:26.9570773Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9570990Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9571198Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9571399Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9571906Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9572110Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9572325Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9572329Z 
2025-07-17T11:07:26.9572419Z SIGABRT(6), PID: 1828345, Thread 1828542: 
2025-07-17T11:07:26.9572889Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9573095Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9573300Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9573569Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9574069Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9574276Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9574577Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9574581Z 
2025-07-17T11:07:26.9574675Z SIGABRT(6), PID: 1828343, Thread 1828407: 
2025-07-17T11:07:26.9575151Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9575417Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9575631Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9575827Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9576388Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9576597Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9576807Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9576811Z 
2025-07-17T11:07:26.9576899Z SIGABRT(6), PID: 1828345, Thread 1828544: 
2025-07-17T11:07:26.9577372Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9577578Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9577783Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9577977Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9578476Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9578771Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9578987Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9578991Z 
2025-07-17T11:07:26.9579082Z SIGABRT(6), PID: 1828343, Thread 1828409: 
2025-07-17T11:07:26.9579557Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9579763Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9579964Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9580164Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9580675Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9580880Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9581087Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9581091Z 
2025-07-17T11:07:26.9581185Z SIGABRT(6), PID: 1828345, Thread 1828546: 
2025-07-17T11:07:26.9581660Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9581952Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9582156Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9582354Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9582925Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9583139Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9583347Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9583425Z 
2025-07-17T11:07:26.9583519Z SIGABRT(6), PID: 1828343, Thread 1828411: 
2025-07-17T11:07:26.9583990Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9584195Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9584395Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9584596Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9585086Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9590334Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9590614Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9590623Z 
2025-07-17T11:07:26.9590731Z SIGABRT(6), PID: 1828345, Thread 1828548: 
2025-07-17T11:07:26.9591234Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9591458Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9591666Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9591991Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9592511Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9592724Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9592934Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9592941Z 
2025-07-17T11:07:26.9593037Z SIGABRT(6), PID: 1828343, Thread 1828413: 
2025-07-17T11:07:26.9593511Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9593724Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9593940Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9594152Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9594672Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9594886Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9595176Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9595271Z SIGABRT(6), PID: 1828345, Thread 1828550: 
2025-07-17T11:07:26.9595748Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9596026Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9596240Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9596451Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9597029Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9597244Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9597456Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9597461Z 
2025-07-17T11:07:26.9597465Z 
2025-07-17T11:07:26.9597560Z SIGABRT(6), PID: 1828343, Thread 1828416: 
2025-07-17T11:07:26.9598028Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9598245Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9598448Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9598656Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9599172Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9599383Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9599592Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9599599Z 
2025-07-17T11:07:26.9599689Z SIGABRT(6), PID: 1828345, Thread 1828552: 
2025-07-17T11:07:26.9600241Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9600457Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9600661Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9600860Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9601366Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9601572Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9601781Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9601785Z 
2025-07-17T11:07:26.9601872Z SIGABRT(6), PID: 1828343, Thread 1828418: 
2025-07-17T11:07:26.9602345Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9602546Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9602749Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9602943Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9603511Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9603718Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9603925Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9604088Z SIGABRT(6), PID: 1828345, Thread 1828554: 
2025-07-17T11:07:26.9604563Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9604770Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9605049Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9605246Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9605744Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9605947Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9606156Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9606162Z 
2025-07-17T11:07:26.9606168Z 
2025-07-17T11:07:26.9606256Z SIGABRT(6), PID: 1828345, Thread 1828556: 
2025-07-17T11:07:26.9606720Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9606922Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9607124Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9607321Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9607811Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9608015Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9608289Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9608294Z 
2025-07-17T11:07:26.9608384Z SIGABRT(6), PID: 1828343, Thread 1828420: 
2025-07-17T11:07:26.9608851Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9609056Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9609259Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9609458Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9609949Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9610162Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9610367Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9610371Z 
2025-07-17T11:07:26.9610460Z SIGABRT(6), PID: 1828345, Thread 1828560: 
2025-07-17T11:07:26.9610925Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9611132Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9611430Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9611625Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9612115Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9612394Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9612600Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9612604Z 
2025-07-17T11:07:26.9612697Z SIGABRT(6), PID: 1828343, Thread 1828422: 
2025-07-17T11:07:26.9613221Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9613433Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9613634Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9613830Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9614317Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9614526Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9614727Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9614731Z 
2025-07-17T11:07:26.9614821Z SIGABRT(6), PID: 1828345, Thread 1828563: 
2025-07-17T11:07:26.9615285Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9615489Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9615687Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9615883Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9616537Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9616754Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9616962Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9616966Z 
2025-07-17T11:07:26.9617059Z SIGABRT(6), PID: 1828343, Thread 1828425: 
2025-07-17T11:07:26.9617529Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9617734Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9617934Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9618134Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9618625Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9618833Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9619040Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9619047Z 
2025-07-17T11:07:26.9619214Z SIGABRT(6), PID: 1828345, Thread 1828566: 
2025-07-17T11:07:26.9619677Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9619880Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9620086Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9620361Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9620856Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9621137Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9621349Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9621356Z 
2025-07-17T11:07:26.9621446Z SIGABRT(6), PID: 1828343, Thread 1828427: 
2025-07-17T11:07:26.9621914Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9622115Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9622319Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9622513Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9623013Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9623218Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9623427Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9623432Z 
2025-07-17T11:07:26.9623518Z SIGABRT(6), PID: 1828345, Thread 1828569: 
2025-07-17T11:07:26.9623982Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9624186Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9624457Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9624655Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9625148Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9625354Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9625561Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9625565Z 
2025-07-17T11:07:26.9625651Z SIGABRT(6), PID: 1828343, Thread 1828429: 
2025-07-17T11:07:26.9626114Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9626319Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9626521Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9626713Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9627208Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9627490Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9627698Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9627702Z 
2025-07-17T11:07:26.9627789Z SIGABRT(6), PID: 1828345, Thread 1828572: 
2025-07-17T11:07:26.9628257Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9628526Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9628730Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9628923Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9629482Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9629692Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9629898Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9629902Z 
2025-07-17T11:07:26.9629989Z SIGABRT(6), PID: 1828343, Thread 1828431: 
2025-07-17T11:07:26.9630459Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9630664Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9630864Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9631065Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9631558Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9631765Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9631970Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9632061Z SIGABRT(6), PID: 1828345, Thread 1828575: 
2025-07-17T11:07:26.9632586Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9632798Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9632998Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9633199Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9633702Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9633906Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9634111Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9634117Z 
2025-07-17T11:07:26.9634121Z 
2025-07-17T11:07:26.9634211Z SIGABRT(6), PID: 1828343, Thread 1828433: 
2025-07-17T11:07:26.9634676Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9634881Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9635082Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9635353Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9635845Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9636051Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9636323Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9636414Z SIGABRT(6), PID: 1828345, Thread 1828579: 
2025-07-17T11:07:26.9636879Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9637142Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9637347Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9637550Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9638043Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9638251Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9638461Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9638465Z 
2025-07-17T11:07:26.9638468Z 
2025-07-17T11:07:26.9638558Z SIGABRT(6), PID: 1828343, Thread 1828436: 
2025-07-17T11:07:26.9639019Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9639226Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9639427Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9639624Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9640117Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9640410Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9640619Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9640713Z SIGABRT(6), PID: 1828345, Thread 1828582: 
2025-07-17T11:07:26.9641183Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9641389Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9641591Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9641785Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9642280Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9642493Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9642707Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9642711Z 
2025-07-17T11:07:26.9642714Z 
2025-07-17T11:07:26.9642805Z SIGABRT(6), PID: 1828345, Thread 1828586: 
2025-07-17T11:07:26.9643276Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9643553Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9643757Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9643951Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9644626Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9644840Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9645050Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9645204Z SIGABRT(6), PID: 1828343, Thread 1828438: 
2025-07-17T11:07:26.9645677Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9645885Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9646087Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9646282Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9646781Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9646987Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9647193Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9647199Z 
2025-07-17T11:07:26.9647203Z 
2025-07-17T11:07:26.9647294Z SIGABRT(6), PID: 1828345, Thread 1828589: 
2025-07-17T11:07:26.9647765Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9647969Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9648172Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9648441Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9648938Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9649144Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9649351Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9649356Z 
2025-07-17T11:07:26.9649447Z SIGABRT(6), PID: 1828343, Thread 1828440: 
2025-07-17T11:07:26.9649915Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9650119Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9650326Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9650522Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9651012Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9651218Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9651425Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9651493Z 
2025-07-17T11:07:26.9651588Z SIGABRT(6), PID: 1828345, Thread 1828593: 
2025-07-17T11:07:26.9652049Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9652255Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9652527Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9652724Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9653277Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9653490Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9653703Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9653707Z 
2025-07-17T11:07:26.9653798Z SIGABRT(6), PID: 1828343, Thread 1828443: 
2025-07-17T11:07:26.9654262Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9654470Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9654672Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9654872Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9655367Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9655577Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9655783Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9655788Z 
2025-07-17T11:07:26.9655881Z SIGABRT(6), PID: 1828345, Thread 1828598: 
2025-07-17T11:07:26.9656489Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9656704Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9656908Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9657107Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9657601Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9657811Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9658015Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9658019Z 
2025-07-17T11:07:26.9658110Z SIGABRT(6), PID: 1828345, Thread 1828601: 
2025-07-17T11:07:26.9658575Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9658781Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9658981Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9659179Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9659675Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9659977Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9660182Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9660187Z 
2025-07-17T11:07:26.9660358Z SIGABRT(6), PID: 1828343, Thread 1828445: 
2025-07-17T11:07:26.9660825Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9661031Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9661233Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9661505Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9662002Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9662213Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9662417Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9662425Z 
2025-07-17T11:07:26.9662515Z SIGABRT(6), PID: 1828345, Thread 1828605: 
2025-07-17T11:07:26.9662983Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9663185Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9663390Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9663587Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9664086Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9664291Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9664572Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9664668Z SIGABRT(6), PID: 1828343, Thread 1828447: 
2025-07-17T11:07:26.9665137Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9665344Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9665547Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9665743Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9666233Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9666439Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9666650Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9666654Z 
2025-07-17T11:07:26.9666657Z 
2025-07-17T11:07:26.9666746Z SIGABRT(6), PID: 1828345, Thread 1828609: 
2025-07-17T11:07:26.9667212Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9667415Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9667690Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9667886Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9668388Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9668669Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9668881Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9668886Z 
2025-07-17T11:07:26.9668976Z SIGABRT(6), PID: 1828343, Thread 1828449: 
2025-07-17T11:07:26.9669534Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9669745Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9669950Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9670154Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9670663Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9670873Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9671085Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9671089Z 
2025-07-17T11:07:26.9671179Z SIGABRT(6), PID: 1828345, Thread 1828614: 
2025-07-17T11:07:26.9671656Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9671862Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9672067Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9672264Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9672829Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9673044Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9673251Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9673255Z 
2025-07-17T11:07:26.9673348Z SIGABRT(6), PID: 1828343, Thread 1828450: 
2025-07-17T11:07:26.9673812Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9674019Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9674220Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9674416Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9674912Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9675123Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9675329Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9675422Z SIGABRT(6), PID: 1828345, Thread 1828620: 
2025-07-17T11:07:26.9675967Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9676175Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9676377Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9676638Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9677136Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9677344Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9677609Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9677614Z 
2025-07-17T11:07:26.9677619Z 
2025-07-17T11:07:26.9677712Z SIGABRT(6), PID: 1828345, Thread 1828625: 
2025-07-17T11:07:26.9678178Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9678385Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9678587Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9678787Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9679283Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9679491Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9679697Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9679702Z 
2025-07-17T11:07:26.9679794Z SIGABRT(6), PID: 1828343, Thread 1828452: 
2025-07-17T11:07:26.9680256Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9680462Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9680729Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9680929Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9681423Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9681633Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9681840Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9681844Z 
2025-07-17T11:07:26.9681940Z SIGABRT(6), PID: 1828345, Thread 1828629: 
2025-07-17T11:07:26.9682404Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9682617Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9682824Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9683021Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9683518Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9683793Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9684005Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9684009Z 
2025-07-17T11:07:26.9684097Z SIGABRT(6), PID: 1828343, Thread 1828454: 
2025-07-17T11:07:26.9684567Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9684837Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9685043Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9685238Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9685791Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9686003Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9686212Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9686218Z 
2025-07-17T11:07:26.9686307Z SIGABRT(6), PID: 1828345, Thread 1828634: 
2025-07-17T11:07:26.9686777Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9686982Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9687185Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9687380Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9687875Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9688081Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9688288Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9688292Z 
2025-07-17T11:07:26.9688381Z SIGABRT(6), PID: 1828343, Thread 1828456: 
2025-07-17T11:07:26.9688920Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9689127Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9689330Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9689527Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9690021Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9690228Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9690435Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9690441Z 
2025-07-17T11:07:26.9690528Z SIGABRT(6), PID: 1828345, Thread 1828638: 
2025-07-17T11:07:26.9690997Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9691203Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9691408Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9691604Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9692170Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9692376Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9692647Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9692653Z 
2025-07-17T11:07:26.9692742Z SIGABRT(6), PID: 1828343, Thread 1828458: 
2025-07-17T11:07:26.9693212Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9693476Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9693681Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9693882Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9694371Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9694580Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9694789Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9694793Z 
2025-07-17T11:07:26.9694883Z SIGABRT(6), PID: 1828345, Thread 1828640: 
2025-07-17T11:07:26.9695349Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9695557Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9695761Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9696013Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9696507Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9696802Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9697018Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9697022Z 
2025-07-17T11:07:26.9697113Z SIGABRT(6), PID: 1828343, Thread 1828460: 
2025-07-17T11:07:26.9697588Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9697797Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9698002Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9698200Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9698696Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9698909Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9699114Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9699118Z 
2025-07-17T11:07:26.9699206Z SIGABRT(6), PID: 1828345, Thread 1828644: 
2025-07-17T11:07:26.9699675Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9699961Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9700165Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9700362Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9700854Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9701162Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9701370Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9701373Z 
2025-07-17T11:07:26.9701542Z SIGABRT(6), PID: 1828343, Thread 1828462: 
2025-07-17T11:07:26.9702014Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9702222Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9702422Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9702617Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9703110Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9703318Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9703523Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9703532Z 
2025-07-17T11:07:26.9703619Z SIGABRT(6), PID: 1828345, Thread 1828648: 
2025-07-17T11:07:26.9704089Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9704291Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9704493Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9704757Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9705256Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9705462Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9705683Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9705688Z 
2025-07-17T11:07:26.9705777Z SIGABRT(6), PID: 1828343, Thread 1828465: 
2025-07-17T11:07:26.9706250Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9706452Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9706658Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9706855Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9707350Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9707555Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9707764Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9707835Z 
2025-07-17T11:07:26.9707925Z SIGABRT(6), PID: 1828345, Thread 1828652: 
2025-07-17T11:07:26.9708390Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9708592Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9708860Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9709055Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9709559Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9709822Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9710035Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9710039Z 
2025-07-17T11:07:26.9710126Z SIGABRT(6), PID: 1828343, Thread 1828467: 
2025-07-17T11:07:26.9710597Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9710801Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9711003Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9711195Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9711692Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9711897Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9712106Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9712110Z 
2025-07-17T11:07:26.9712197Z SIGABRT(6), PID: 1828345, Thread 1828656: 
2025-07-17T11:07:26.9712665Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9712935Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9713145Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9713338Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9713833Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9714038Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9714246Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9714250Z 
2025-07-17T11:07:26.9714337Z SIGABRT(6), PID: 1828343, Thread 1828469: 
2025-07-17T11:07:26.9714802Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9715008Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9715209Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9715406Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9715897Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9716172Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9716381Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9716385Z 
2025-07-17T11:07:26.9716475Z SIGABRT(6), PID: 1828345, Thread 1828660: 
2025-07-17T11:07:26.9717012Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9717219Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9717419Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9717683Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9718184Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9718391Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9718598Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9718603Z 
2025-07-17T11:07:26.9718694Z SIGABRT(6), PID: 1828343, Thread 1828471: 
2025-07-17T11:07:26.9719159Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9719367Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9719567Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9719765Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9720260Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9720469Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9720676Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9720683Z 
2025-07-17T11:07:26.9720840Z SIGABRT(6), PID: 1828345, Thread 1828664: 
2025-07-17T11:07:26.9721306Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9721512Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9721717Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9721915Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9722407Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9722613Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9722820Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9722824Z 
2025-07-17T11:07:26.9722916Z SIGABRT(6), PID: 1828343, Thread 1828473: 
2025-07-17T11:07:26.9723380Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9723589Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9723863Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9724063Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9724556Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9724842Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9725049Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9725055Z 
2025-07-17T11:07:26.9725143Z SIGABRT(6), PID: 1828345, Thread 1828668: 
2025-07-17T11:07:26.9725678Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9725886Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9726094Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9726286Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9726779Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9726988Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9727197Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9727201Z 
2025-07-17T11:07:26.9727289Z SIGABRT(6), PID: 1828343, Thread 1828475: 
2025-07-17T11:07:26.9727762Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9727965Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9728169Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9728364Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9728922Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9729133Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9729340Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9729344Z 
2025-07-17T11:07:26.9729432Z SIGABRT(6), PID: 1828345, Thread 1828672: 
2025-07-17T11:07:26.9729902Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9730108Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9730311Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9730503Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9731002Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9731207Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9731419Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9731423Z 
2025-07-17T11:07:26.9731512Z SIGABRT(6), PID: 1828343, Thread 1828477: 
2025-07-17T11:07:26.9731983Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9732278Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9732483Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9732743Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9733242Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9733447Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9733709Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9733714Z 
2025-07-17T11:07:26.9733806Z SIGABRT(6), PID: 1828345, Thread 1828677: 
2025-07-17T11:07:26.9734281Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9734488Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9734694Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9734893Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9735387Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9735591Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9735804Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9735810Z 
2025-07-17T11:07:26.9735952Z SIGABRT(6), PID: 1828343, Thread 1828479: 
2025-07-17T11:07:26.9736420Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9736625Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9736913Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9737115Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9737606Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9737815Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9738022Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9738026Z 
2025-07-17T11:07:26.9738116Z SIGABRT(6), PID: 1828345, Thread 1828682: 
2025-07-17T11:07:26.9738581Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9738804Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9739413Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9739963Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9740788Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9741596Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9742195Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9742516Z 
2025-07-17T11:07:26.9742613Z SIGABRT(6), PID: 1828343, Thread 1828481: 
2025-07-17T11:07:26.9743320Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9744237Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9744832Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9745341Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9746257Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9747090Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9747603Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9747912Z 
2025-07-17T11:07:26.9748020Z SIGABRT(6), PID: 1828345, Thread 1828687: 
2025-07-17T11:07:26.9748700Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9749467Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9750104Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9750609Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9751406Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9752200Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9752712Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9753023Z 
2025-07-17T11:07:26.9753115Z SIGABRT(6), PID: 1828343, Thread 1828483: 
2025-07-17T11:07:26.9753828Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9754593Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9755103Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9755605Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9756404Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9757198Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9757706Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9758011Z 
2025-07-17T11:07:26.9758104Z SIGABRT(6), PID: 1828343, Thread 1828485: 
2025-07-17T11:07:26.9758750Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9759513Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9760020Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9760516Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9761377Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9762169Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9762679Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9763050Z 
2025-07-17T11:07:26.9763145Z SIGABRT(6), PID: 1828345, Thread 1828692: 
2025-07-17T11:07:26.9763789Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9764547Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9765107Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9765617Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9766404Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9767202Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9767717Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9768020Z 
2025-07-17T11:07:26.9768116Z SIGABRT(6), PID: 1828343, Thread 1828487: 
2025-07-17T11:07:26.9768761Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9769525Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9770025Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9770517Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9771307Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9772095Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9772674Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9772983Z 
2025-07-17T11:07:26.9773077Z SIGABRT(6), PID: 1828345, Thread 1828696: 
2025-07-17T11:07:26.9773719Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9774481Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9774984Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9775480Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9776341Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9777132Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9777639Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9777944Z 
2025-07-17T11:07:26.9778033Z SIGABRT(6), PID: 1828343, Thread 1828489: 
2025-07-17T11:07:26.9778682Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9779532Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9780035Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9780530Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9781319Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9782189Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9782701Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9783008Z 
2025-07-17T11:07:26.9783099Z SIGABRT(6), PID: 1828345, Thread 1828699: 
2025-07-17T11:07:26.9783817Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9784585Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9785087Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9785580Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9786372Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9787161Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9787670Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9787974Z 
2025-07-17T11:07:26.9788068Z SIGABRT(6), PID: 1828343, Thread 1828491: 
2025-07-17T11:07:26.9788709Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9789473Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9789969Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9790462Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9791357Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9792149Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9792658Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9792963Z 
2025-07-17T11:07:26.9793056Z SIGABRT(6), PID: 1828345, Thread 1828704: 
2025-07-17T11:07:26.9793698Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9794454Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9794956Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9795455Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9796237Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9797026Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9797535Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9797915Z 
2025-07-17T11:07:26.9798006Z SIGABRT(6), PID: 1828343, Thread 1828493: 
2025-07-17T11:07:26.9798650Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9799408Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9799979Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9800479Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9801259Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9802109Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9802623Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9802934Z 
2025-07-17T11:07:26.9803024Z SIGABRT(6), PID: 1828345, Thread 1828708: 
2025-07-17T11:07:26.9803666Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9804427Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9804932Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9805427Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9806217Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9807005Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9807515Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9807819Z 
2025-07-17T11:07:26.9807910Z SIGABRT(6), PID: 1828343, Thread 1828495: 
2025-07-17T11:07:26.9808557Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9809390Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9809895Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9810392Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9811182Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9811974Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9812485Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9812791Z 
2025-07-17T11:07:26.9812884Z SIGABRT(6), PID: 1828345, Thread 1828712: 
2025-07-17T11:07:26.9813524Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9814287Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9814787Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9815280Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9816134Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9817130Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9817638Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9817943Z 
2025-07-17T11:07:26.9818033Z SIGABRT(6), PID: 1828343, Thread 1828497: 
2025-07-17T11:07:26.9818752Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9819511Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9820010Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9820581Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9821373Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9822168Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9822675Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9822980Z 
2025-07-17T11:07:26.9823072Z SIGABRT(6), PID: 1828345, Thread 1828716: 
2025-07-17T11:07:26.9823718Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9824474Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9824977Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9825479Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9826269Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9827057Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9827565Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9827872Z 
2025-07-17T11:07:26.9828045Z SIGABRT(6), PID: 1828343, Thread 1828499: 
2025-07-17T11:07:26.9828692Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9829449Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9829952Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9830450Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9831238Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9832032Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9832544Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9832850Z 
2025-07-17T11:07:26.9832943Z SIGABRT(6), PID: 1828345, Thread 1828720: 
2025-07-17T11:07:26.9833583Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9834341Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9834844Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9835431Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9836217Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9837010Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9837591Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9837900Z 
2025-07-17T11:07:26.9837990Z SIGABRT(6), PID: 1828343, Thread 1828501: 
2025-07-17T11:07:26.9838634Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9839453Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9839962Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9840456Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9841245Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9842038Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9842545Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9842853Z 
2025-07-17T11:07:26.9842943Z SIGABRT(6), PID: 1828345, Thread 1828724: 
2025-07-17T11:07:26.9843592Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9844353Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9844854Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9845354Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9846143Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9847006Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9847519Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9847824Z 
2025-07-17T11:07:26.9847917Z SIGABRT(6), PID: 1828343, Thread 1828503: 
2025-07-17T11:07:26.9848563Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9849328Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9849829Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9850319Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9851103Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9851899Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9852411Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9852713Z 
2025-07-17T11:07:26.9852809Z SIGABRT(6), PID: 1828345, Thread 1828728: 
2025-07-17T11:07:26.9853452Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9854283Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9854784Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9855276Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9856208Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9856999Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9857508Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9857919Z 
2025-07-17T11:07:26.9858015Z SIGABRT(6), PID: 1828343, Thread 1828505: 
2025-07-17T11:07:26.9858661Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9859418Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9859920Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9860418Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9861203Z frame #4: <unknown function> + 0x3475ab (0x7f90ff2425ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9861991Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9862500Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9862809Z 
2025-07-17T11:07:26.9862899Z SIGABRT(6), PID: 1828345, Thread 1828732: 
2025-07-17T11:07:26.9863537Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9864296Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9864798Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9865375Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9866172Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9866962Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9867467Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9867778Z 
2025-07-17T11:07:26.9867870Z SIGABRT(6), PID: 1828343, Thread 1829400: 
2025-07-17T11:07:26.9868508Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9869512Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f9010aa008a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9870251Z frame #2: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9870742Z frame #3: pthread_kill + 0x12c (0x7f91023ac9fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9871192Z frame #4: raise + 0x16 (0x7f9102358476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9871607Z frame #5: abort + 0xd3 (0x7f910233e7f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9872096Z frame #6: <unknown function> + 0x972cc (0x7f8f7e6292cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9872745Z frame #7: <unknown function> + 0x95f47 (0x7f8f7e627f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9873296Z frame #8: <unknown function> + 0x34b71 (0x7f8f7e5c6b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9873823Z frame #9: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9874434Z frame #10: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9874747Z 
2025-07-17T11:07:26.9874842Z SIGABRT(6), PID: 1828345, Thread 1828736: 
2025-07-17T11:07:26.9875492Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9876317Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9876830Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9877334Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9878134Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9878928Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9879443Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9879756Z 
2025-07-17T11:07:26.9879850Z SIGABRT(6), PID: 1828343, Thread 1831925: 
2025-07-17T11:07:26.9880496Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9881255Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9881719Z frame #2: __poll + 0x4f (0x7f910242ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9882382Z frame #3: <unknown function> + 0x489f (0x7f9101f3a89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:26.9883107Z frame #4: <unknown function> + 0x1483bc (0x559fcb4bf3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9883752Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9884325Z frame #6: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9884887Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9885451Z frame #8: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9886024Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x559fcb4a7a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9886599Z frame #10: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9887175Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x559fcb4a5c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9887745Z frame #12: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9888312Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9888877Z frame #14: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9889438Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9889991Z frame #16: <unknown function> + 0x147734 (0x559fcb4be734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9890528Z frame #17: <unknown function> + 0x21ca82 (0x559fcb593a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9891138Z frame #18: <unknown function> + 0x21c9e4 (0x559fcb5939e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9891666Z frame #19: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9892193Z frame #20: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9892577Z 
2025-07-17T11:07:26.9892671Z SIGABRT(6), PID: 1828345, Thread 1828740: 
2025-07-17T11:07:26.9893331Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9894107Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9894677Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9895189Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9896089Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9896890Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9897407Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9897718Z 
2025-07-17T11:07:26.9897815Z SIGABRT(6), PID: 1828345, Thread 1828744: 
2025-07-17T11:07:26.9898462Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9899225Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9899735Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9900236Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9901034Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9901829Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9902426Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9902745Z 
2025-07-17T11:07:26.9902842Z SIGABRT(6), PID: 1828345, Thread 1828748: 
2025-07-17T11:07:26.9903490Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9904253Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9904754Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9905255Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9906049Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9906837Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9907346Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9907653Z 
2025-07-17T11:07:26.9907744Z SIGABRT(6), PID: 1828343, Thread 1831927: 
2025-07-17T11:07:26.9908389Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9909149Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9909742Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9910265Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f91023aa2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9911030Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f903c7e8048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9912064Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f903c7eabf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9912802Z frame #6: <unknown function> + 0xdc253 (0x7f900ed0c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9913338Z frame #7: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9913926Z frame #8: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9914331Z SIGABRT(6), PID: 1828345, Thread 1828756: 
2025-07-17T11:07:26.9914972Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9915733Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9916235Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9916737Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9917533Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9918324Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9918837Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9919144Z 
2025-07-17T11:07:26.9919148Z 
2025-07-17T11:07:26.9919242Z SIGABRT(6), PID: 1828345, Thread 1828762: 
2025-07-17T11:07:26.9919881Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9920635Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9921203Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9921705Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9922489Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9923284Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9923792Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9924094Z 
2025-07-17T11:07:26.9924189Z SIGABRT(6), PID: 1828343, Thread 1831928: 
2025-07-17T11:07:26.9924829Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9925587Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9926088Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9926608Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f91023aa2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9927390Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f903c7e2741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:26.9928251Z frame #5: <unknown function> + 0xdc253 (0x7f900ed0c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:26.9928784Z frame #6: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9929290Z frame #7: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9929599Z 
2025-07-17T11:07:26.9929690Z SIGABRT(6), PID: 1828345, Thread 1828767: 
2025-07-17T11:07:26.9930427Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9931187Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9931689Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9932248Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9933047Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9933838Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9934347Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9934658Z 
2025-07-17T11:07:26.9934748Z SIGABRT(6), PID: 1828343, Thread 1832013: 
2025-07-17T11:07:26.9935387Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9936209Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9936670Z frame #2: ioctl + 0x3f (0x7f910243094f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9937163Z frame #3: <unknown function> + 0x13eef0 (0x7f8f7e6d0ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9937726Z frame #4: <unknown function> + 0x1378c4 (0x7f8f7e6c98c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9938283Z frame #5: <unknown function> + 0x95b2b (0x7f8f7e627b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9938836Z frame #6: <unknown function> + 0x34b71 (0x7f8f7e5c6b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9939367Z frame #7: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9939966Z frame #8: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9940284Z 
2025-07-17T11:07:26.9940381Z SIGABRT(6), PID: 1828345, Thread 1828770: 
2025-07-17T11:07:26.9941034Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9941801Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9942307Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9942806Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9943602Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9944396Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9944907Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9945218Z 
2025-07-17T11:07:26.9945309Z SIGABRT(6), PID: 1828345, Thread 1828773: 
2025-07-17T11:07:26.9945958Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9946799Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9947302Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9947795Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9948591Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9949460Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9949965Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9950272Z 
2025-07-17T11:07:26.9950365Z SIGABRT(6), PID: 1828343, Thread 1833159: 
2025-07-17T11:07:26.9951086Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9951855Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9952316Z frame #2: __poll + 0x4f (0x7f910242ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9952783Z frame #3: <unknown function> + 0xd4540 (0x7f8fc6840540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9953298Z frame #4: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9953811Z frame #5: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9954205Z SIGABRT(6), PID: 1828345, Thread 1828776: 
2025-07-17T11:07:26.9954844Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9955602Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9956107Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9956601Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9957392Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9958253Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9958775Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9959079Z 
2025-07-17T11:07:26.9959082Z 
2025-07-17T11:07:26.9959175Z SIGABRT(6), PID: 1828343, Thread 1833166: 
2025-07-17T11:07:26.9959817Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9960576Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9961037Z frame #2: __poll + 0x4f (0x7f910242ebcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9961505Z frame #3: <unknown function> + 0xd65a2 (0x7f8fc68425a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9962013Z frame #4: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9962524Z frame #5: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9962832Z 
2025-07-17T11:07:26.9962921Z SIGABRT(6), PID: 1828345, Thread 1828779: 
2025-07-17T11:07:26.9963560Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9964319Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9964900Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9965401Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9966189Z frame #4: <unknown function> + 0x3475ab (0x7f6024cf75ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:26.9967057Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9967565Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9967871Z 
2025-07-17T11:07:26.9967962Z SIGABRT(6), PID: 1828343, Thread 1833169: 
2025-07-17T11:07:26.9968669Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9969427Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9969928Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9970420Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9970923Z frame #4: <unknown function> + 0xd1e7f (0x7f8fc683de7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:26.9971441Z frame #5: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9971945Z frame #6: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9972247Z 
2025-07-17T11:07:26.9972339Z SIGABRT(6), PID: 1828345, Thread 1829409: 
2025-07-17T11:07:26.9972978Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9973975Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f5f3655608a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9974717Z frame #2: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9975211Z frame #3: pthread_kill + 0x12c (0x7f6027e619fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9975655Z frame #4: raise + 0x16 (0x7f6027e0d476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9976219Z frame #5: abort + 0xd3 (0x7f6027df37f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9976726Z frame #6: <unknown function> + 0x972cc (0x7f5ea40df2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9977284Z frame #7: <unknown function> + 0x95f47 (0x7f5ea40ddf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9984110Z frame #8: <unknown function> + 0x34b71 (0x7f5ea407cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:26.9984739Z frame #9: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9985280Z frame #10: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9985596Z 
2025-07-17T11:07:26.9985697Z SIGABRT(6), PID: 1828343, Thread 1833178: 
2025-07-17T11:07:26.9986397Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:26.9987192Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9987706Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9988213Z frame #3: <unknown function> + 0x9ca53 (0x7f91023b2a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9988775Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x559fcb48bbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9989357Z frame #5: <unknown function> + 0x1d314f (0x559fcb54a14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9990053Z frame #6: <unknown function> + 0x1d2f6b (0x559fcb549f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9990596Z frame #7: <unknown function> + 0x6df36 (0x559fcb3e4f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9991151Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9991798Z frame #9: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9992352Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9992912Z frame #11: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9993469Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9994102Z frame #13: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9994668Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9995220Z frame #15: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9995777Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9996325Z frame #17: <unknown function> + 0x147734 (0x559fcb4be734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9996855Z frame #18: <unknown function> + 0x21ca82 (0x559fcb593a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9997386Z frame #19: <unknown function> + 0x21c9e4 (0x559fcb5939e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:26.9997912Z frame #20: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9998433Z frame #21: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:26.9998837Z SIGABRT(6), PID: 1828345, Thread 1832126: 
2025-07-17T11:07:26.9999495Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0000266Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0000733Z frame #2: __poll + 0x4f (0x7f6027ee3bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0001474Z frame #3: <unknown function> + 0x489f (0x7f60279ef89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:27.0002201Z frame #4: <unknown function> + 0x1483bc (0x563858c863bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0002759Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0003323Z frame #6: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0003880Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0004427Z frame #8: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0004982Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x563858c6ea65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0005551Z frame #10: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0006114Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x563858c6cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0006676Z frame #12: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0007224Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0007776Z frame #14: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0008425Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0008967Z frame #16: <unknown function> + 0x147734 (0x563858c85734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0009491Z frame #17: <unknown function> + 0x21ca82 (0x563858d5aa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0010022Z frame #18: <unknown function> + 0x21c9e4 (0x563858d5a9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0010624Z frame #19: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0011146Z frame #20: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0011453Z 
2025-07-17T11:07:27.0011458Z 
2025-07-17T11:07:27.0011552Z SIGABRT(6), PID: 1828343, Thread 1833190: 
2025-07-17T11:07:27.0012274Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0013050Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0013511Z frame #2: read + 0x4c (0x7f910242a81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0013939Z frame #3: _Py_read + 0x4c (0x559fcb551a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0014431Z frame #4: <unknown function> + 0x208c3b (0x559fcb57fc3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0014975Z frame #5: <unknown function> + 0x14648d (0x559fcb4bd48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0015537Z frame #6: PyObject_VectorcallMethod + 0x85 (0x559fcb4cc215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0016196Z frame #7: <unknown function> + 0x208ae7 (0x559fcb57fae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0016728Z frame #8: <unknown function> + 0x208472 (0x559fcb57f472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0017261Z frame #9: <unknown function> + 0x1da6ee (0x559fcb5516ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0017797Z frame #10: <unknown function> + 0x1483bc (0x559fcb4bf3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0018357Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0018911Z frame #12: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0019559Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x559fcb4a33bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0020117Z frame #14: <unknown function> + 0x147734 (0x559fcb4be734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0020677Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x559fcb4a5c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0021245Z frame #16: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0021791Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0022342Z frame #18: _PyFunction_Vectorcall + 0x6c (0x559fcb4b30ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0022888Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x559fcb4a37b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0023437Z frame #20: <unknown function> + 0x147734 (0x559fcb4be734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0023970Z frame #21: <unknown function> + 0x21ca82 (0x559fcb593a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0024497Z frame #22: <unknown function> + 0x21c9e4 (0x559fcb5939e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0025019Z frame #23: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0025537Z frame #24: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0025842Z 
2025-07-17T11:07:27.0026041Z SIGABRT(6), PID: 1828345, Thread 1832127: 
2025-07-17T11:07:27.0026722Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0027498Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0028004Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0028615Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6027e5f2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0029381Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f5f6229e048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0030423Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f5f622a0bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0031199Z frame #6: <unknown function> + 0xdc253 (0x7f5f347c2253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0031741Z frame #7: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0032254Z frame #8: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0032563Z 
2025-07-17T11:07:27.0032661Z SIGABRT(6), PID: 1828345, Thread 1832128: 
2025-07-17T11:07:27.0033322Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0034086Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0034589Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0035109Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f6027e5f2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0035894Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f5f62298741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0036677Z frame #5: <unknown function> + 0xdc253 (0x7f5f347c2253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0037202Z frame #6: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0037716Z frame #7: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0038093Z 
2025-07-17T11:07:27.0038192Z SIGABRT(6), PID: 1828345, Thread 1832131: 
2025-07-17T11:07:27.0038836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0039594Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0040057Z frame #2: ioctl + 0x3f (0x7f6027ee594f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0040550Z frame #3: <unknown function> + 0x13eef0 (0x7f5ea4186ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0041109Z frame #4: <unknown function> + 0x1378c4 (0x7f5ea417f8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0041661Z frame #5: <unknown function> + 0x95b2b (0x7f5ea40ddb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0042209Z frame #6: <unknown function> + 0x34b71 (0x7f5ea407cb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0042736Z frame #7: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0043246Z frame #8: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0043555Z 
2025-07-17T11:07:27.0043645Z SIGABRT(6), PID: 1828345, Thread 1833155: 
2025-07-17T11:07:27.0044292Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0045122Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0045587Z frame #2: __poll + 0x4f (0x7f6027ee3bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0046057Z frame #3: <unknown function> + 0xd4540 (0x7f5eec2f6540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0046574Z frame #4: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0047161Z frame #5: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0047467Z 
2025-07-17T11:07:27.0047560Z SIGABRT(6), PID: 1828343, Thread 1833216: 
2025-07-17T11:07:27.0048201Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0049020Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0049530Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0049733Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0050112Z frame #4: <unknown function> + 0x5a02023 (0x7f90467d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0050685Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f90467da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0051279Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f90467d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0051668Z frame #7: <unknown function> + 0x79d655 (0x7f9057cc1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0051909Z frame #8: <unknown function> + 0xdc253 (0x7f900ed0c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0052123Z frame #9: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0052344Z frame #10: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0052351Z 
2025-07-17T11:07:27.0052444Z SIGABRT(6), PID: 1828345, Thread 1833165: 
2025-07-17T11:07:27.0052984Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0053197Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0053370Z frame #2: __poll + 0x4f (0x7f6027ee3bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0053593Z frame #3: <unknown function> + 0xd65a2 (0x7f5eec2f85a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0053805Z frame #4: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0054015Z frame #5: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0054022Z 
2025-07-17T11:07:27.0054112Z SIGABRT(6), PID: 1828345, Thread 1833171: 
2025-07-17T11:07:27.0054590Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0054799Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0055003Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0055206Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0055424Z frame #4: <unknown function> + 0xd1e7f (0x7f5eec2f3e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0055702Z frame #5: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0055975Z frame #6: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0055979Z 
2025-07-17T11:07:27.0056071Z SIGABRT(6), PID: 1828345, Thread 1833181: 
2025-07-17T11:07:27.0056543Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0056840Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0057044Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0057247Z frame #3: <unknown function> + 0x9ca53 (0x7f6027e67a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0057573Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x563858c52bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0057809Z frame #5: <unknown function> + 0x1d314f (0x563858d1114f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0058034Z frame #6: <unknown function> + 0x1d2f6b (0x563858d10f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0058256Z frame #7: <unknown function> + 0x6df36 (0x563858babf36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0058501Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0058728Z frame #9: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0058970Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0059198Z frame #11: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0059436Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0059663Z frame #13: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0059897Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0060120Z frame #15: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0060354Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0060674Z frame #17: <unknown function> + 0x147734 (0x563858c85734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0060899Z frame #18: <unknown function> + 0x21ca82 (0x563858d5aa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0061117Z frame #19: <unknown function> + 0x21c9e4 (0x563858d5a9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0061332Z frame #20: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0061547Z frame #21: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0061552Z 
2025-07-17T11:07:27.0061647Z SIGABRT(6), PID: 1828345, Thread 1833193: 
2025-07-17T11:07:27.0062123Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0062337Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0062503Z frame #2: read + 0x4c (0x7f6027edf81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0062686Z frame #3: _Py_read + 0x4c (0x563858d18a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0062908Z frame #4: <unknown function> + 0x208c3b (0x563858d46c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0063127Z frame #5: <unknown function> + 0x14648d (0x563858c8448d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0063370Z frame #6: PyObject_VectorcallMethod + 0x85 (0x563858c93215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0063674Z frame #7: <unknown function> + 0x208ae7 (0x563858d46ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0063888Z frame #8: <unknown function> + 0x208472 (0x563858d46472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0064106Z frame #9: <unknown function> + 0x1da6ee (0x563858d186ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0064396Z frame #10: <unknown function> + 0x1483bc (0x563858c863bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0064637Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0064861Z frame #12: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0065158Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x563858c6a3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0065378Z frame #14: <unknown function> + 0x147734 (0x563858c85734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0065631Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x563858c6cc55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0065855Z frame #16: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0066089Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0066313Z frame #18: _PyFunction_Vectorcall + 0x6c (0x563858c7a0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0066553Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x563858c6a7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0066774Z frame #20: <unknown function> + 0x147734 (0x563858c85734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0066998Z frame #21: <unknown function> + 0x21ca82 (0x563858d5aa82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0067215Z frame #22: <unknown function> + 0x21c9e4 (0x563858d5a9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0067433Z frame #23: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0067644Z frame #24: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0067648Z 
2025-07-17T11:07:27.0067743Z SIGABRT(6), PID: 1828343, Thread 1833217: 
2025-07-17T11:07:27.0068288Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0068505Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0068712Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0068918Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0069305Z frame #4: <unknown function> + 0x5a02023 (0x7f90467d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0069882Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f90467da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0070477Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f90467d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0070865Z frame #7: <unknown function> + 0x79d655 (0x7f9057cc1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0071096Z frame #8: <unknown function> + 0xdc253 (0x7f900ed0c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0071311Z frame #9: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0071590Z frame #10: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0071595Z 
2025-07-17T11:07:27.0071689Z SIGABRT(6), PID: 1828342, Thread 1828342: 
2025-07-17T11:07:27.0072180Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0072398Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0072702Z frame #2: <unknown function> + 0x7741f (0x7fec4e7d241f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0072935Z frame #3: <unknown function> + 0x76c8e (0x7fec4e7d1c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0073162Z frame #4: <unknown function> + 0x6ad69 (0x7fec4e7c5d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0073448Z frame #5: <unknown function> + 0x3f54db (0x7fecdf42a4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0073670Z frame #6: <unknown function> + 0x3e11d8 (0x7fecdf4161d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0073886Z frame #7: <unknown function> + 0xb2f86 (0x7fecdf0e7f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0074094Z frame #8: <unknown function> + 0x8e501 (0x7fecdf0c3501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0074489Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7fed09eca7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:27.0074882Z frame #10: <unknown function> + 0xd307d9 (0x7fed2841d7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0075111Z frame #11: <unknown function> + 0x139902 (0x559d851bb902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0075365Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x559d851b2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0075597Z frame #13: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0075840Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x559d851b2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0076068Z frame #15: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0076307Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0076732Z frame #17: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0076974Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0077197Z frame #19: <unknown function> + 0x1474b2 (0x559d851c94b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0077400Z frame #20: PyObject_Call + 0xbc (0x559d851c9e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0077624Z frame #21: <unknown function> + 0x1ff69d (0x559d8528169d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0077847Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x559d851b7023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0078094Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x559d851b2cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0078319Z frame #24: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0078559Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x559d851ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0078785Z frame #26: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0079022Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x559d851ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0079243Z frame #28: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0079487Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x559d851b2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0079814Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x559d851b64b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0080040Z frame #31: _PyObject_Call_Prepend + 0x69 (0x559d851c7ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0080257Z frame #32: <unknown function> + 0x205949 (0x559d85287949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0080483Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x559d851b7023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0080790Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x559d851b2cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0081016Z frame #35: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0081251Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x559d851ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0081537Z frame #37: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0081779Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x559d851ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0082003Z frame #39: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0082237Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x559d851ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0082461Z frame #41: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0082704Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x559d851af3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0082927Z frame #43: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0083317Z frame #44: <unknown function> + 0x7ab3f7 (0x7fed27e983f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0083540Z frame #45: <unknown function> + 0x13bc88 (0x559d851bdc88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0083745Z frame #46: PyObject_Call + 0x20f (0x559d851c9f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0083989Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x559d851b398d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0084202Z frame #48: <unknown function> + 0x147608 (0x559d851c9608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0084445Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x559d851b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0084736Z frame #50: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0084979Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x559d851ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0085201Z frame #52: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0085443Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x559d851af3f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0085667Z frame #54: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0085908Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x559d851ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0086128Z frame #56: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0086369Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x559d851b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0086594Z frame #58: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0086833Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x559d851b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0087051Z frame #60: <unknown function> + 0x147608 (0x559d851c9608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0087291Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x559d851b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0087574Z frame #62: <unknown function> + 0x147608 (0x559d851c9608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0087578Z 
2025-07-17T11:07:27.0087674Z SIGABRT(6), PID: 1828342, Thread 1828350: 
2025-07-17T11:07:27.0088154Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0088437Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0088648Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0088856Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0089450Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0089666Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0089884Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0089888Z 
2025-07-17T11:07:27.0089982Z SIGABRT(6), PID: 1828345, Thread 1833245: 
2025-07-17T11:07:27.0090457Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0090668Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0090875Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0091077Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0091454Z frame #4: <unknown function> + 0x5a02023 (0x7f5f6c28c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0092018Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f6c2901ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0092609Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f6c2871d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0093059Z frame #7: <unknown function> + 0x79d655 (0x7f5f7d777655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0093297Z frame #8: <unknown function> + 0xdc253 (0x7f5f347c2253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0093506Z frame #9: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0093724Z frame #10: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0093730Z 
2025-07-17T11:07:27.0093821Z SIGABRT(6), PID: 1828342, Thread 1828351: 
2025-07-17T11:07:27.0094298Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0094504Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0094711Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0094912Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0095418Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0095627Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0095849Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0095968Z 
2025-07-17T11:07:27.0096064Z SIGABRT(6), PID: 1828342, Thread 1828352: 
2025-07-17T11:07:27.0096539Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0096744Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0097036Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0097231Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0097735Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0098017Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0098235Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0098324Z SIGABRT(6), PID: 1828343, Thread 1833218: 
2025-07-17T11:07:27.0098796Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0099001Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0099207Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0099404Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0099780Z frame #4: <unknown function> + 0x5a02023 (0x7f90467d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0100344Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f90467da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0100938Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f90467d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0101396Z frame #7: <unknown function> + 0x79d655 (0x7f9057cc1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0101629Z frame #8: <unknown function> + 0xdc253 (0x7f900ed0c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0101839Z frame #9: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0102055Z frame #10: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0102059Z 
2025-07-17T11:07:27.0102063Z 
2025-07-17T11:07:27.0102157Z SIGABRT(6), PID: 1828342, Thread 1828353: 
2025-07-17T11:07:27.0102627Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0102835Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0103036Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0103238Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0103736Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0103945Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0104155Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0104234Z 
2025-07-17T11:07:27.0104329Z SIGABRT(6), PID: 1828342, Thread 1828354: 
2025-07-17T11:07:27.0104798Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0105003Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0105273Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0105477Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0105979Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0106253Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0106469Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0106474Z 
2025-07-17T11:07:27.0106566Z SIGABRT(6), PID: 1828345, Thread 1833246: 
2025-07-17T11:07:27.0107035Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0107243Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0107451Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0107650Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0108021Z frame #4: <unknown function> + 0x5a02023 (0x7f5f6c28c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0108583Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f6c2901ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0109171Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f6c2871d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0109617Z frame #7: <unknown function> + 0x79d655 (0x7f5f7d777655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0109853Z frame #8: <unknown function> + 0xdc253 (0x7f5f347c2253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0110066Z frame #9: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0110279Z frame #10: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0110288Z 
2025-07-17T11:07:27.0110376Z SIGABRT(6), PID: 1828342, Thread 1828355: 
2025-07-17T11:07:27.0110854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0111062Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0111267Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0111466Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0111969Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0112177Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0112390Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0112460Z 
2025-07-17T11:07:27.0112551Z SIGABRT(6), PID: 1828342, Thread 1828356: 
2025-07-17T11:07:27.0113027Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0113232Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0113436Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0113697Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0114195Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0114462Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0114677Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0114684Z 
2025-07-17T11:07:27.0114773Z SIGABRT(6), PID: 1828343, Thread 1833219: 
2025-07-17T11:07:27.0115238Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0115439Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0115660Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0115853Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0116223Z frame #4: <unknown function> + 0x5a02023 (0x7f90467d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0116782Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f90467da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0117371Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f90467d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0117748Z frame #7: <unknown function> + 0x79d655 (0x7f9057cc1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0118046Z frame #8: <unknown function> + 0xdc253 (0x7f900ed0c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0118257Z frame #9: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0118472Z frame #10: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0118476Z 
2025-07-17T11:07:27.0118566Z SIGABRT(6), PID: 1828342, Thread 1828357: 
2025-07-17T11:07:27.0119038Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0119246Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0119452Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0119651Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0120155Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0120360Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0120572Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0120576Z 
2025-07-17T11:07:27.0120668Z SIGABRT(6), PID: 1828342, Thread 1828358: 
2025-07-17T11:07:27.0121216Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0121428Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0121631Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0121927Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0122432Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0122649Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0122923Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0122930Z 
2025-07-17T11:07:27.0123025Z SIGABRT(6), PID: 1828342, Thread 1828359: 
2025-07-17T11:07:27.0123500Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0123709Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0123915Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0124119Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0124620Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0124833Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0125045Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0125050Z 
2025-07-17T11:07:27.0125141Z SIGABRT(6), PID: 1828345, Thread 1833248: 
2025-07-17T11:07:27.0125607Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0125814Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0126080Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0126282Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0126659Z frame #4: <unknown function> + 0x5a02023 (0x7f5f6c28c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0127224Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f6c2901ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0127815Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f6c2871d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0128200Z frame #7: <unknown function> + 0x79d655 (0x7f5f7d777655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0128434Z frame #8: <unknown function> + 0xdc253 (0x7f5f347c2253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0128646Z frame #9: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0128860Z frame #10: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0128865Z 
2025-07-17T11:07:27.0128957Z SIGABRT(6), PID: 1828342, Thread 1828360: 
2025-07-17T11:07:27.0129497Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0129708Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0129910Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0130175Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0130683Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0130893Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0131164Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0131169Z 
2025-07-17T11:07:27.0131262Z SIGABRT(6), PID: 1828342, Thread 1828361: 
2025-07-17T11:07:27.0131740Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0131945Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0132149Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0132351Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0132851Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0133057Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0133271Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0133276Z 
2025-07-17T11:07:27.0133365Z SIGABRT(6), PID: 1828343, Thread 1833220: 
2025-07-17T11:07:27.0133837Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0134041Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0134310Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0134509Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0134883Z frame #4: <unknown function> + 0x5a02023 (0x7f90467d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0135446Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f90467da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0136126Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f90467d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0136506Z frame #7: <unknown function> + 0x79d655 (0x7f9057cc1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0136745Z frame #8: <unknown function> + 0xdc253 (0x7f900ed0c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0136951Z frame #9: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0137164Z frame #10: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0137168Z 
2025-07-17T11:07:27.0137258Z SIGABRT(6), PID: 1828342, Thread 1828362: 
2025-07-17T11:07:27.0137733Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0138034Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0138245Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0138446Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0139041Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0139250Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0139466Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0139541Z 
2025-07-17T11:07:27.0139634Z SIGABRT(6), PID: 1828342, Thread 1828363: 
2025-07-17T11:07:27.0140114Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0140323Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0140526Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0140731Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0141228Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0141440Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0141651Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0141656Z 
2025-07-17T11:07:27.0141747Z SIGABRT(6), PID: 1828342, Thread 1828364: 
2025-07-17T11:07:27.0142215Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0142423Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0142629Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0142908Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0143405Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0143618Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0143827Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0143833Z 
2025-07-17T11:07:27.0143923Z SIGABRT(6), PID: 1828345, Thread 1833250: 
2025-07-17T11:07:27.0144392Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0144604Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0144810Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0145008Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0145384Z frame #4: <unknown function> + 0x5a02023 (0x7f5f6c28c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0145951Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f6c2901ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0146617Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f6c2871d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0146999Z frame #7: <unknown function> + 0x79d655 (0x7f5f7d777655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0147301Z frame #8: <unknown function> + 0xdc253 (0x7f5f347c2253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0147515Z frame #9: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0147732Z frame #10: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0147736Z 
2025-07-17T11:07:27.0147888Z SIGABRT(6), PID: 1828342, Thread 1828365: 
2025-07-17T11:07:27.0148362Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0148578Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0148781Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0148986Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0149493Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0149702Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0149911Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0149918Z 
2025-07-17T11:07:27.0150006Z SIGABRT(6), PID: 1828342, Thread 1828366: 
2025-07-17T11:07:27.0150478Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0150683Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0150886Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0151147Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0151651Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0151859Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0152074Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0152079Z 
2025-07-17T11:07:27.0152168Z SIGABRT(6), PID: 1828343, Thread 1833221: 
2025-07-17T11:07:27.0152635Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0152839Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0153045Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0153246Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0153619Z frame #4: <unknown function> + 0x5a02023 (0x7f90467d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0154181Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f90467da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0154856Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f90467d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0155235Z frame #7: <unknown function> + 0x79d655 (0x7f9057cc1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0155532Z frame #8: <unknown function> + 0xdc253 (0x7f900ed0c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0155751Z frame #9: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0155967Z frame #10: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0155972Z 
2025-07-17T11:07:27.0156061Z SIGABRT(6), PID: 1828342, Thread 1828367: 
2025-07-17T11:07:27.0156593Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0156805Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0157013Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0157213Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0157717Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0157926Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0158137Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0158141Z 
2025-07-17T11:07:27.0158231Z SIGABRT(6), PID: 1828342, Thread 1828368: 
2025-07-17T11:07:27.0158704Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0158911Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0159115Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0159312Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0159873Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0160085Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0160298Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0160302Z 
2025-07-17T11:07:27.0160395Z SIGABRT(6), PID: 1828342, Thread 1828369: 
2025-07-17T11:07:27.0160870Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0161078Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0161280Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0161482Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0161975Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0162186Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0162393Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0162462Z 
2025-07-17T11:07:27.0162555Z SIGABRT(6), PID: 1828345, Thread 1833251: 
2025-07-17T11:07:27.0163022Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0163232Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0163504Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0163703Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0164077Z frame #4: <unknown function> + 0x5a02023 (0x7f5f6c28c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0164705Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f6c2901ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0165304Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f6c2871d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0165688Z frame #7: <unknown function> + 0x79d655 (0x7f5f7d777655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0165922Z frame #8: <unknown function> + 0xdc253 (0x7f5f347c2253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0166134Z frame #9: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0166348Z frame #10: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0166352Z 
2025-07-17T11:07:27.0166446Z SIGABRT(6), PID: 1828342, Thread 1828370: 
2025-07-17T11:07:27.0166916Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0167126Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0167327Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0167528Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0168096Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0168310Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0168519Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0168525Z 
2025-07-17T11:07:27.0168616Z SIGABRT(6), PID: 1828342, Thread 1828371: 
2025-07-17T11:07:27.0169090Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0169295Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0169499Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0169704Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0170198Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0170408Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0170615Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0170622Z 
2025-07-17T11:07:27.0170775Z SIGABRT(6), PID: 1828343, Thread 1833222: 
2025-07-17T11:07:27.0171242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0171446Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0171652Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0171916Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0172290Z frame #4: <unknown function> + 0x5a02023 (0x7f90467d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0172912Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f90467da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0173509Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f90467d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0173890Z frame #7: <unknown function> + 0x79d655 (0x7f9057cc1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0174127Z frame #8: <unknown function> + 0xdc253 (0x7f900ed0c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0174334Z frame #9: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0174551Z frame #10: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0174555Z 
2025-07-17T11:07:27.0174643Z SIGABRT(6), PID: 1828342, Thread 1828372: 
2025-07-17T11:07:27.0175119Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0175327Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0175533Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0175732Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0176391Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0176613Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0176828Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0176833Z 
2025-07-17T11:07:27.0176924Z SIGABRT(6), PID: 1828342, Thread 1828373: 
2025-07-17T11:07:27.0177406Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0177614Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0177822Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0178021Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0178527Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0178736Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0178948Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0178953Z 
2025-07-17T11:07:27.0179042Z SIGABRT(6), PID: 1828342, Thread 1828374: 
2025-07-17T11:07:27.0179607Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0179812Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0180018Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0180296Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0180797Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0181005Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0181285Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0181292Z 
2025-07-17T11:07:27.0181385Z SIGABRT(6), PID: 1828345, Thread 1833253: 
2025-07-17T11:07:27.0181854Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0182061Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0182264Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0182467Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0182837Z frame #4: <unknown function> + 0x5a02023 (0x7f5f6c28c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0183401Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f6c2901ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0183992Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f6c2871d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0184372Z frame #7: <unknown function> + 0x79d655 (0x7f5f7d777655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0184688Z frame #8: <unknown function> + 0xdc253 (0x7f5f347c2253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0184903Z frame #9: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0185121Z frame #10: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0185125Z 
2025-07-17T11:07:27.0185216Z SIGABRT(6), PID: 1828342, Thread 1828376: 
2025-07-17T11:07:27.0185687Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0185896Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0186097Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0186296Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0186797Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0187006Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0187217Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0187222Z 
2025-07-17T11:07:27.0187314Z SIGABRT(6), PID: 1828342, Thread 1828377: 
2025-07-17T11:07:27.0187780Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0188060Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0188262Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0188522Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0189024Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0189232Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0189498Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0189502Z 
2025-07-17T11:07:27.0189599Z SIGABRT(6), PID: 1828343, Thread 1833223: 
2025-07-17T11:07:27.0190072Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9010a9fb9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0190277Z frame #1: <unknown function> + 0x42520 (0x7f9102358520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0190480Z frame #2: <unknown function> + 0x91117 (0x7f91023a7117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0190677Z frame #3: pthread_cond_wait + 0x211 (0x7f91023a9a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0191050Z frame #4: <unknown function> + 0x5a02023 (0x7f90467d6023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0191613Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f90467da1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0192203Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f90467d11d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0192584Z frame #7: <unknown function> + 0x79d655 (0x7f9057cc1655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0192815Z frame #8: <unknown function> + 0xdc253 (0x7f900ed0c253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0193088Z frame #9: <unknown function> + 0x94ac3 (0x7f91023aaac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0193310Z frame #10: <unknown function> + 0x126850 (0x7f910243c850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0193315Z 
2025-07-17T11:07:27.0193404Z SIGABRT(6), PID: 1828342, Thread 1828379: 
2025-07-17T11:07:27.0193879Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0194085Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0194292Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0194488Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0194988Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0195197Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0195407Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0195413Z 
2025-07-17T11:07:27.0195503Z SIGABRT(6), PID: 1828342, Thread 1828380: 
2025-07-17T11:07:27.0195971Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0196246Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0196452Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0196653Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0197220Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0197425Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0197637Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0197700Z 
2025-07-17T11:07:27.0197793Z SIGABRT(6), PID: 1828342, Thread 1828382: 
2025-07-17T11:07:27.0198272Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0198475Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0198682Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0198884Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0199385Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0199592Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0199803Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0199808Z 
2025-07-17T11:07:27.0199897Z SIGABRT(6), PID: 1828345, Thread 1833254: 
2025-07-17T11:07:27.0200367Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0200574Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0200775Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0201037Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0201417Z frame #4: <unknown function> + 0x5a02023 (0x7f5f6c28c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0201982Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f6c2901ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0202569Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f6c2871d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0202952Z frame #7: <unknown function> + 0x79d655 (0x7f5f7d777655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0203182Z frame #8: <unknown function> + 0xdc253 (0x7f5f347c2253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0203392Z frame #9: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0203607Z frame #10: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0203610Z 
2025-07-17T11:07:27.0203704Z SIGABRT(6), PID: 1828342, Thread 1828384: 
2025-07-17T11:07:27.0204172Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0204448Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0204651Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0204850Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0205411Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0205622Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0205831Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0205835Z 
2025-07-17T11:07:27.0206095Z SIGABRT(6), PID: 1828342, Thread 1828386: 
2025-07-17T11:07:27.0206568Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0206778Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0206981Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0207184Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0207681Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0207889Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0208099Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0208103Z 
2025-07-17T11:07:27.0208194Z SIGABRT(6), PID: 1828342, Thread 1828387: 
2025-07-17T11:07:27.0208665Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0208875Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0209076Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0209338Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0209833Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0210041Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0210246Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0210254Z 
2025-07-17T11:07:27.0210343Z SIGABRT(6), PID: 1828342, Thread 1828390: 
2025-07-17T11:07:27.0210809Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0211012Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0211218Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0211416Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0211912Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0212118Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0212329Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0212398Z 
2025-07-17T11:07:27.0212488Z SIGABRT(6), PID: 1828342, Thread 1828391: 
2025-07-17T11:07:27.0212959Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0213161Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0213431Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0213627Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0214121Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0214410Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0214630Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0214719Z SIGABRT(6), PID: 1828345, Thread 1833255: 
2025-07-17T11:07:27.0215185Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f5f36555b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0215391Z frame #1: <unknown function> + 0x42520 (0x7f6027e0d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0215597Z frame #2: <unknown function> + 0x91117 (0x7f6027e5c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0215796Z frame #3: pthread_cond_wait + 0x211 (0x7f6027e5ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0216221Z frame #4: <unknown function> + 0x5a02023 (0x7f5f6c28c023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0216785Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f5f6c2901ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0217383Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f5f6c2871d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0217842Z frame #7: <unknown function> + 0x79d655 (0x7f5f7d777655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0218081Z frame #8: <unknown function> + 0xdc253 (0x7f5f347c2253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0218291Z frame #9: <unknown function> + 0x94ac3 (0x7f6027e5fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0218504Z frame #10: <unknown function> + 0x126850 (0x7f6027ef1850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0218510Z 
2025-07-17T11:07:27.0218513Z 
2025-07-17T11:07:27.0218602Z SIGABRT(6), PID: 1828342, Thread 1828393: 
2025-07-17T11:07:27.0219084Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0219293Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0219499Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0219702Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0220203Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0220408Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0220618Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0220707Z 
2025-07-17T11:07:27.0220800Z SIGABRT(6), PID: 1828342, Thread 1828395: 
2025-07-17T11:07:27.0221270Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0221478Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0221763Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0221969Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0222465Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0222763Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0222991Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0222995Z 
2025-07-17T11:07:27.0223090Z SIGABRT(6), PID: 1828342, Thread 1828397: 
2025-07-17T11:07:27.0223580Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0223789Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0224003Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0224206Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0224714Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0224926Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0225135Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0225140Z 
2025-07-17T11:07:27.0225234Z SIGABRT(6), PID: 1828342, Thread 1828399: 
2025-07-17T11:07:27.0225708Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0225984Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0226194Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0226396Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0226900Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0227114Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0227323Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0227327Z 
2025-07-17T11:07:27.0227419Z SIGABRT(6), PID: 1828342, Thread 1828401: 
2025-07-17T11:07:27.0227892Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0228106Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0228308Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0228507Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0229003Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0229292Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0229502Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0229506Z 
2025-07-17T11:07:27.0229599Z SIGABRT(6), PID: 1828342, Thread 1828404: 
2025-07-17T11:07:27.0230137Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0230346Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0230551Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0230808Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0231307Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0231522Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0231731Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0231739Z 
2025-07-17T11:07:27.0231827Z SIGABRT(6), PID: 1828342, Thread 1828406: 
2025-07-17T11:07:27.0232299Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0232505Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0232716Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0232910Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0233407Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0233614Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0233824Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0233831Z 
2025-07-17T11:07:27.0233985Z SIGABRT(6), PID: 1828342, Thread 1828408: 
2025-07-17T11:07:27.0234460Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0234665Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0234878Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0235075Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0235575Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0235782Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0235995Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0235999Z 
2025-07-17T11:07:27.0236087Z SIGABRT(6), PID: 1828342, Thread 1828410: 
2025-07-17T11:07:27.0236561Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0236767Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0236977Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0237239Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0237735Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0237947Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0238246Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0238250Z 
2025-07-17T11:07:27.0238351Z SIGABRT(6), PID: 1828342, Thread 1828412: 
2025-07-17T11:07:27.0238828Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0239102Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0239322Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0239528Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0240035Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0240249Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0240466Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0240470Z 
2025-07-17T11:07:27.0240565Z SIGABRT(6), PID: 1828342, Thread 1828414: 
2025-07-17T11:07:27.0241060Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0241277Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0241488Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0241686Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0242249Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0242470Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0242677Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0242681Z 
2025-07-17T11:07:27.0242779Z SIGABRT(6), PID: 1828342, Thread 1828415: 
2025-07-17T11:07:27.0243250Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0243462Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0243663Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0243872Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0244375Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0244593Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0244806Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0244810Z 
2025-07-17T11:07:27.0244912Z SIGABRT(6), PID: 1828342, Thread 1828417: 
2025-07-17T11:07:27.0245391Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0245681Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0245884Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0246090Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0246678Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0246900Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0247167Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0247172Z 
2025-07-17T11:07:27.0247282Z SIGABRT(6), PID: 1828342, Thread 1828419: 
2025-07-17T11:07:27.0247771Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0247994Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0248203Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0248414Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0248919Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0249138Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0249345Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0249350Z 
2025-07-17T11:07:27.0249442Z SIGABRT(6), PID: 1828342, Thread 1828421: 
2025-07-17T11:07:27.0249908Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0250125Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0250336Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0250612Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0251117Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0251337Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0251554Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0251561Z 
2025-07-17T11:07:27.0251669Z SIGABRT(6), PID: 1828342, Thread 1828423: 
2025-07-17T11:07:27.0252145Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0252365Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0252570Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0252768Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0253264Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0253470Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0253764Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0253768Z 
2025-07-17T11:07:27.0253861Z SIGABRT(6), PID: 1828342, Thread 1828424: 
2025-07-17T11:07:27.0254349Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0254616Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0254830Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0255032Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0255608Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0255827Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0256098Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0256103Z 
2025-07-17T11:07:27.0256191Z SIGABRT(6), PID: 1828342, Thread 1828426: 
2025-07-17T11:07:27.0256661Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0256876Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0257089Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0257285Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0257781Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0257991Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0258200Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0258204Z 
2025-07-17T11:07:27.0258291Z SIGABRT(6), PID: 1828342, Thread 1828428: 
2025-07-17T11:07:27.0258847Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0259053Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0259260Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0259460Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0259959Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0260166Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0260376Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0260380Z 
2025-07-17T11:07:27.0260469Z SIGABRT(6), PID: 1828342, Thread 1828430: 
2025-07-17T11:07:27.0260937Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0261142Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0261348Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0261545Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0262124Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0262330Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0262538Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0262614Z 
2025-07-17T11:07:27.0262709Z SIGABRT(6), PID: 1828342, Thread 1828432: 
2025-07-17T11:07:27.0263181Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0263388Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0263659Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0263861Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0264362Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0264571Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0264781Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0264786Z 
2025-07-17T11:07:27.0264878Z SIGABRT(6), PID: 1828342, Thread 1828434: 
2025-07-17T11:07:27.0265345Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0265553Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0265754Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0265957Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0266451Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0266664Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0266938Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0266942Z 
2025-07-17T11:07:27.0267037Z SIGABRT(6), PID: 1828342, Thread 1828435: 
2025-07-17T11:07:27.0267507Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0267716Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0267921Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0268118Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0268614Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0268825Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0269032Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0269036Z 
2025-07-17T11:07:27.0269127Z SIGABRT(6), PID: 1828342, Thread 1828437: 
2025-07-17T11:07:27.0269592Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0269799Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0270069Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0270268Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0270763Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0271037Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0271248Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0271252Z 
2025-07-17T11:07:27.0271346Z SIGABRT(6), PID: 1828342, Thread 1828439: 
2025-07-17T11:07:27.0271875Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0272088Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0272290Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0272488Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0272980Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0273193Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0273401Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0273407Z 
2025-07-17T11:07:27.0273497Z SIGABRT(6), PID: 1828342, Thread 1828441: 
2025-07-17T11:07:27.0273972Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0274177Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0274381Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0274580Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0275150Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0275360Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0275569Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0275575Z 
2025-07-17T11:07:27.0275664Z SIGABRT(6), PID: 1828342, Thread 1828442: 
2025-07-17T11:07:27.0276141Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0276346Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0276548Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0276749Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0277246Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0277452Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0277667Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0277755Z 
2025-07-17T11:07:27.0277846Z SIGABRT(6), PID: 1828342, Thread 1828444: 
2025-07-17T11:07:27.0278320Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0278524Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0278730Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0278998Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0279499Z frame #4: <unknown function> + 0x3475ab (0x7fedcf40b5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0279768Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0279985Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0279992Z 
2025-07-17T11:07:27.0280080Z SIGABRT(6), PID: 1828342, Thread 1829398: 
2025-07-17T11:07:27.0280554Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0281004Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7fece0c6908a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0281217Z frame #2: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0281410Z frame #3: pthread_kill + 0x12c (0x7fedd25759fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0281580Z frame #4: raise + 0x16 (0x7fedd2521476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0281740Z frame #5: abort + 0xd3 (0x7fedd25077f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0281980Z frame #6: <unknown function> + 0x972cc (0x7fec4e7f22cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0282213Z frame #7: <unknown function> + 0x95f47 (0x7fec4e7f0f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0282440Z frame #8: <unknown function> + 0x34b71 (0x7fec4e78fb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0282649Z frame #9: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0282932Z frame #10: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0282937Z 
2025-07-17T11:07:27.0283031Z SIGABRT(6), PID: 1828342, Thread 1831726: 
2025-07-17T11:07:27.0283512Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0283718Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0283889Z frame #2: __poll + 0x4f (0x7fedd25f7bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0284301Z frame #3: <unknown function> + 0x489f (0x7fedd210389f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:27.0284527Z frame #4: <unknown function> + 0x1483bc (0x559d851ca3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0284770Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0284998Z frame #6: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0285233Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0285457Z frame #8: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0285703Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x559d851b2a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0285934Z frame #10: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0286253Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x559d851b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0286482Z frame #12: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0286719Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0287005Z frame #14: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0287239Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0287460Z frame #16: <unknown function> + 0x147734 (0x559d851c9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0287741Z frame #17: <unknown function> + 0x21ca82 (0x559d8529ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0287966Z frame #18: <unknown function> + 0x21c9e4 (0x559d8529e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0288188Z frame #19: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0288400Z frame #20: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0288404Z 
2025-07-17T11:07:27.0288495Z SIGABRT(6), PID: 1828344, Thread 1828344: 
2025-07-17T11:07:27.0288971Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0289182Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0289415Z frame #2: <unknown function> + 0x773e7 (0x7f9dc75f33e7 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0289642Z frame #3: <unknown function> + 0x76c8e (0x7f9dc75f2c8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0289866Z frame #4: <unknown function> + 0x6ad69 (0x7f9dc75e6d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0290085Z frame #5: <unknown function> + 0x3f54db (0x7f9e5824b4db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0290296Z frame #6: <unknown function> + 0x3e11d8 (0x7f9e582371d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0290512Z frame #7: <unknown function> + 0xb2f86 (0x7f9e57f08f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0290788Z frame #8: <unknown function> + 0x8e501 (0x7f9e57ee4501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0291176Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f9e82ceb7ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:27.0291563Z frame #10: <unknown function> + 0xd307d9 (0x7f9ea123a7d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0291790Z frame #11: <unknown function> + 0x139902 (0x555cf89c2902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0292046Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x555cf89b9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0292272Z frame #13: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0292515Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x555cf89b9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0292740Z frame #15: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0292984Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0293210Z frame #17: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0293448Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0293671Z frame #19: <unknown function> + 0x1474b2 (0x555cf89d04b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0293947Z frame #20: PyObject_Call + 0xbc (0x555cf89d0e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0294170Z frame #21: <unknown function> + 0x1ff69d (0x555cf8a8869d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0294399Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x555cf89be023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0294639Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x555cf89b9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0294928Z frame #24: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0295164Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x555cf89b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0295387Z frame #26: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0295684Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x555cf89b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0295960Z frame #28: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0296202Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x555cf89b9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0296461Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x555cf89bd4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0296680Z frame #31: _PyObject_Call_Prepend + 0x69 (0x555cf89ceab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0296904Z frame #32: <unknown function> + 0x205949 (0x555cf8a8e949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0297129Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x555cf89be023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0297369Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x555cf89b9cc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0297596Z frame #35: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0297839Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x555cf89b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0298063Z frame #37: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0298300Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x555cf89b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0298522Z frame #39: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0298847Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x555cf89b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0299073Z frame #41: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0299314Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x555cf89b63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0299538Z frame #43: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0299931Z frame #44: <unknown function> + 0x7ab3f7 (0x7f9ea0cb53f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0300154Z frame #45: <unknown function> + 0x13bc88 (0x555cf89c4c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0300365Z frame #46: PyObject_Call + 0x20f (0x555cf89d0f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0300605Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x555cf89ba98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0300829Z frame #48: <unknown function> + 0x147608 (0x555cf89d0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0301066Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x555cf89b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0301293Z frame #50: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0301531Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x555cf89b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0301833Z frame #52: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0302071Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x555cf89b63f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0302299Z frame #54: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0302534Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x555cf89b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0302840Z frame #56: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0303081Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x555cf89b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0303307Z frame #58: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0303617Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x555cf89b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0303844Z frame #60: <unknown function> + 0x147608 (0x555cf89d0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0304081Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x555cf89b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0304300Z frame #62: <unknown function> + 0x147608 (0x555cf89d0608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0304304Z 
2025-07-17T11:07:27.0304395Z SIGABRT(6), PID: 1828342, Thread 1831727: 
2025-07-17T11:07:27.0304885Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0305099Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0305284Z frame #2: epoll_wait + 0x5e (0x7fedd2604e2e in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0305621Z frame #3: uv__io_poll + 0x6bd (0x7fed231bf44d in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0305944Z frame #4: uv_run + 0x12a (0x7fed231b336a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0306322Z frame #5: <unknown function> + 0x658a08a (0x7fed1752708a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0306554Z frame #6: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0306852Z frame #7: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0307069Z frame #8: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0307074Z 
2025-07-17T11:07:27.0307167Z SIGABRT(6), PID: 1828344, Thread 1828446: 
2025-07-17T11:07:27.0307640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0307851Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0308053Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0308260Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0308764Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0308976Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0309184Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0309188Z 
2025-07-17T11:07:27.0309280Z SIGABRT(6), PID: 1828344, Thread 1828448: 
2025-07-17T11:07:27.0309748Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0310026Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0310230Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0310428Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0310989Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0311199Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0311408Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0311412Z 
2025-07-17T11:07:27.0311565Z SIGABRT(6), PID: 1828342, Thread 1833139: 
2025-07-17T11:07:27.0312036Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0312246Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0312447Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0312676Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fedd25732dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0313121Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7fed0c9b1048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0313549Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7fed0c9b3bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0313783Z frame #6: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0313998Z frame #7: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0314209Z frame #8: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0314213Z 
2025-07-17T11:07:27.0314305Z SIGABRT(6), PID: 1828344, Thread 1828451: 
2025-07-17T11:07:27.0314769Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0315044Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0315255Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0315456Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0315952Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0316162Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0316366Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0316373Z 
2025-07-17T11:07:27.0316461Z SIGABRT(6), PID: 1828342, Thread 1833140: 
2025-07-17T11:07:27.0316933Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0317140Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0317343Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0317569Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fedd25732dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0318046Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7fed0c9ab741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0318350Z frame #5: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0318564Z frame #6: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0318770Z frame #7: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0318834Z 
2025-07-17T11:07:27.0318930Z SIGABRT(6), PID: 1828344, Thread 1828453: 
2025-07-17T11:07:27.0319399Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0319609Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0319872Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0320079Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0320573Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0320780Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0320990Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0321080Z SIGABRT(6), PID: 1828342, Thread 1833142: 
2025-07-17T11:07:27.0321547Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0321757Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0321927Z frame #2: ioctl + 0x3f (0x7fedd25f994f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0322168Z frame #3: <unknown function> + 0x13eef0 (0x7fec4e899ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0322401Z frame #4: <unknown function> + 0x1378c4 (0x7fec4e8928c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0322630Z frame #5: <unknown function> + 0x95b2b (0x7fec4e7f0b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0322858Z frame #6: <unknown function> + 0x34b71 (0x7fec4e78fb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0323135Z frame #7: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0323346Z frame #8: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0323353Z 
2025-07-17T11:07:27.0323356Z 
2025-07-17T11:07:27.0323448Z SIGABRT(6), PID: 1828344, Thread 1828455: 
2025-07-17T11:07:27.0323919Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0324127Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0324328Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0324529Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0325029Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0325236Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0325443Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0325447Z 
2025-07-17T11:07:27.0325539Z SIGABRT(6), PID: 1828342, Thread 1833158: 
2025-07-17T11:07:27.0326006Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0326282Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0326449Z frame #2: __poll + 0x4f (0x7fedd25f7bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0326664Z frame #3: <unknown function> + 0xd4540 (0x7fec96a09540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0326935Z frame #4: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0327144Z frame #5: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0327149Z 
2025-07-17T11:07:27.0327238Z SIGABRT(6), PID: 1828342, Thread 1833167: 
2025-07-17T11:07:27.0327765Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0327975Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0328139Z frame #2: __poll + 0x4f (0x7fedd25f7bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0328354Z frame #3: <unknown function> + 0xd65a2 (0x7fec96a0b5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0328561Z frame #4: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0328780Z frame #5: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0328784Z 
2025-07-17T11:07:27.0328878Z SIGABRT(6), PID: 1828344, Thread 1828457: 
2025-07-17T11:07:27.0329339Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0329546Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0329751Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0329948Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0330438Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0330707Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0330919Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0331010Z SIGABRT(6), PID: 1828342, Thread 1833170: 
2025-07-17T11:07:27.0331479Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0331685Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0331892Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0332088Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0332303Z frame #4: <unknown function> + 0xd1e7f (0x7fec96a06e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0332507Z frame #5: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0332721Z frame #6: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0332725Z 
2025-07-17T11:07:27.0332728Z 
2025-07-17T11:07:27.0332818Z SIGABRT(6), PID: 1828344, Thread 1828459: 
2025-07-17T11:07:27.0333290Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0333493Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0333766Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0333960Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0334462Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0334849Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0335058Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0335064Z 
2025-07-17T11:07:27.0335152Z SIGABRT(6), PID: 1828342, Thread 1833180: 
2025-07-17T11:07:27.0335717Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0335988Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0336204Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0336409Z frame #3: <unknown function> + 0x9ca53 (0x7fedd257ba53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0336665Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x559d85196bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0336894Z frame #5: <unknown function> + 0x1d314f (0x559d8525514f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0337116Z frame #6: <unknown function> + 0x1d2f6b (0x559d85254f6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0337334Z frame #7: <unknown function> + 0x6df36 (0x559d850eff36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0337582Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0337807Z frame #9: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0338050Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0338276Z frame #11: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0338518Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0338830Z frame #13: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0339071Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0339297Z frame #15: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0339533Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0339763Z frame #17: <unknown function> + 0x147734 (0x559d851c9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0339985Z frame #18: <unknown function> + 0x21ca82 (0x559d8529ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0340203Z frame #19: <unknown function> + 0x21c9e4 (0x559d8529e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0340417Z frame #20: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0340631Z frame #21: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0340636Z 
2025-07-17T11:07:27.0340727Z SIGABRT(6), PID: 1828344, Thread 1828461: 
2025-07-17T11:07:27.0341198Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0341412Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0341698Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0341905Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0342401Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0342692Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0342905Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0342908Z 
2025-07-17T11:07:27.0343001Z SIGABRT(6), PID: 1828342, Thread 1833197: 
2025-07-17T11:07:27.0343541Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0343755Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0343925Z frame #2: read + 0x4c (0x7fedd25f381c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0344109Z frame #3: _Py_read + 0x4c (0x559d8525ca6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0344334Z frame #4: <unknown function> + 0x208c3b (0x559d8528ac3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0344558Z frame #5: <unknown function> + 0x14648d (0x559d851c848d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0344808Z frame #6: PyObject_VectorcallMethod + 0x85 (0x559d851d7215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0345029Z frame #7: <unknown function> + 0x208ae7 (0x559d8528aae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0345244Z frame #8: <unknown function> + 0x208472 (0x559d8528a472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0345466Z frame #9: <unknown function> + 0x1da6ee (0x559d8525c6ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0345689Z frame #10: <unknown function> + 0x1483bc (0x559d851ca3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0345930Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0346152Z frame #12: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0346397Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x559d851ae3bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0346681Z frame #14: <unknown function> + 0x147734 (0x559d851c9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0346935Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x559d851b0c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0347161Z frame #16: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0347397Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0347623Z frame #18: _PyFunction_Vectorcall + 0x6c (0x559d851be0ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0347859Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x559d851ae7b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0348076Z frame #20: <unknown function> + 0x147734 (0x559d851c9734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0348297Z frame #21: <unknown function> + 0x21ca82 (0x559d8529ea82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0348520Z frame #22: <unknown function> + 0x21c9e4 (0x559d8529e9e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0348739Z frame #23: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0348952Z frame #24: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0348956Z 
2025-07-17T11:07:27.0349049Z SIGABRT(6), PID: 1828344, Thread 1828463: 
2025-07-17T11:07:27.0349524Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0349806Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0350008Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0350211Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0350778Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0350993Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0351198Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0351264Z 
2025-07-17T11:07:27.0351358Z SIGABRT(6), PID: 1828344, Thread 1828464: 
2025-07-17T11:07:27.0351836Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0352042Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0352247Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0352448Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0352947Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0353153Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0353363Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0353369Z 
2025-07-17T11:07:27.0353457Z SIGABRT(6), PID: 1828344, Thread 1828466: 
2025-07-17T11:07:27.0353925Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0354129Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0354334Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0354604Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0355103Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0355314Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0355521Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0355527Z 
2025-07-17T11:07:27.0355615Z SIGABRT(6), PID: 1828344, Thread 1828468: 
2025-07-17T11:07:27.0356094Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0356298Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0356507Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0356703Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0357198Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0357406Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0357683Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0357688Z 
2025-07-17T11:07:27.0357776Z SIGABRT(6), PID: 1828344, Thread 1828470: 
2025-07-17T11:07:27.0358245Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0358515Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0364135Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0364404Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0365067Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0365309Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0365544Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0365551Z 
2025-07-17T11:07:27.0365654Z SIGABRT(6), PID: 1828344, Thread 1828472: 
2025-07-17T11:07:27.0366160Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0366383Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0366590Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0366791Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0367307Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0367516Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0367726Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0367731Z 
2025-07-17T11:07:27.0367823Z SIGABRT(6), PID: 1828342, Thread 1833239: 
2025-07-17T11:07:27.0368371Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0368592Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0368797Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0369001Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0369379Z frame #4: <unknown function> + 0x5a02023 (0x7fed1699f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0369952Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed169a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0370543Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed1699a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0370931Z frame #7: <unknown function> + 0x79d655 (0x7fed27e8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0371167Z frame #8: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0371381Z frame #9: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0371596Z frame #10: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0371664Z 
2025-07-17T11:07:27.0371760Z SIGABRT(6), PID: 1828344, Thread 1828474: 
2025-07-17T11:07:27.0372230Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0372439Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0372734Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0372934Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0373499Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0373711Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0373921Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0373926Z 
2025-07-17T11:07:27.0374016Z SIGABRT(6), PID: 1828344, Thread 1828476: 
2025-07-17T11:07:27.0374481Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0374690Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0374889Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0375083Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0375580Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0375787Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0376079Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0376084Z 
2025-07-17T11:07:27.0376177Z SIGABRT(6), PID: 1828344, Thread 1828478: 
2025-07-17T11:07:27.0376637Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0376929Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0377135Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0377331Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0377824Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0378031Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0378236Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0378244Z 
2025-07-17T11:07:27.0378332Z SIGABRT(6), PID: 1828344, Thread 1828480: 
2025-07-17T11:07:27.0378801Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0379004Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0379206Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0379400Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0379892Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0380183Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0380393Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0380398Z 
2025-07-17T11:07:27.0380566Z SIGABRT(6), PID: 1828344, Thread 1828482: 
2025-07-17T11:07:27.0381033Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0381233Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0381435Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0381709Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0382217Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0382421Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0382631Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0382637Z 
2025-07-17T11:07:27.0382727Z SIGABRT(6), PID: 1828344, Thread 1828484: 
2025-07-17T11:07:27.0383203Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0383404Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0383608Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0383805Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0384296Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0384497Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0384774Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0384779Z 
2025-07-17T11:07:27.0384870Z SIGABRT(6), PID: 1828342, Thread 1833241: 
2025-07-17T11:07:27.0385340Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0385544Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0385748Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0385946Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0386335Z frame #4: <unknown function> + 0x5a02023 (0x7fed1699f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0386901Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed169a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0387493Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed1699a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0387877Z frame #7: <unknown function> + 0x79d655 (0x7fed27e8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0388115Z frame #8: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0388392Z frame #9: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0388612Z frame #10: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0388617Z 
2025-07-17T11:07:27.0388707Z SIGABRT(6), PID: 1828344, Thread 1828486: 
2025-07-17T11:07:27.0389239Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0389446Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0389644Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0389900Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0390400Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0390609Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0390815Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0390823Z 
2025-07-17T11:07:27.0390916Z SIGABRT(6), PID: 1828346, Thread 1828346: 
2025-07-17T11:07:27.0391378Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0391584Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0391817Z frame #2: <unknown function> + 0x7741f (0x7f41c74ce41f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0392044Z frame #3: <unknown function> + 0x76c8e (0x7f41c74cdc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0392268Z frame #4: <unknown function> + 0x6ad69 (0x7f41c74c1d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0392482Z frame #5: <unknown function> + 0x3f54db (0x7f42581264db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0392686Z frame #6: <unknown function> + 0x3e11d8 (0x7f42581121d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0392899Z frame #7: <unknown function> + 0xb2f86 (0x7f4257de3f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0393166Z frame #8: <unknown function> + 0x8e501 (0x7f4257dbf501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0393543Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f4282bc67ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:27.0393922Z frame #10: <unknown function> + 0xd307d9 (0x7f42a11197d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0394152Z frame #11: <unknown function> + 0x139902 (0x556877b45902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0394400Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x556877b3ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0394626Z frame #13: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0394867Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x556877b3ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0395097Z frame #15: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0395335Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0395555Z frame #17: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0395787Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0396005Z frame #19: <unknown function> + 0x1474b2 (0x556877b534b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0396277Z frame #20: PyObject_Call + 0xbc (0x556877b53e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0396499Z frame #21: <unknown function> + 0x1ff69d (0x556877c0b69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0396722Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x556877b41023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0397031Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x556877b3ccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0397255Z frame #24: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0397489Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x556877b383bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0397770Z frame #26: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0398009Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x556877b383bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0398228Z frame #28: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0398466Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x556877b3ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0398716Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x556877b404b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0398933Z frame #31: _PyObject_Call_Prepend + 0x69 (0x556877b51ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0399147Z frame #32: <unknown function> + 0x205949 (0x556877c11949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0399368Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x556877b41023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0399603Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x556877b3ccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0399826Z frame #35: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0400057Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x556877b383bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0400276Z frame #37: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0400507Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x556877b383bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0400792Z frame #39: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0401025Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x556877b383bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0401246Z frame #41: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0401485Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x556877b393f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0401707Z frame #43: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0402095Z frame #44: <unknown function> + 0x7ab3f7 (0x7f42a0b943f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0402312Z frame #45: <unknown function> + 0x13bc88 (0x556877b47c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0402513Z frame #46: PyObject_Call + 0x20f (0x556877b53f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0402758Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x556877b3d98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0402974Z frame #48: <unknown function> + 0x147608 (0x556877b53608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0403210Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x556877b3ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0403430Z frame #50: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0403662Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x556877b383bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0403967Z frame #52: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0404202Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x556877b393f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0404421Z frame #54: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0404731Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x556877b383bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0404949Z frame #56: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0405187Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x556877b3ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0405461Z frame #58: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0405703Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x556877b3ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0405919Z frame #60: <unknown function> + 0x147608 (0x556877b53608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0406155Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x556877b3ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0406368Z frame #62: <unknown function> + 0x147608 (0x556877b53608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0406374Z 
2025-07-17T11:07:27.0406472Z SIGABRT(6), PID: 1828344, Thread 1828488: 
2025-07-17T11:07:27.0406946Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0407159Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0407364Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0407564Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0408059Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0408267Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0408473Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0408546Z 
2025-07-17T11:07:27.0408643Z SIGABRT(6), PID: 1828344, Thread 1828490: 
2025-07-17T11:07:27.0409117Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0409324Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0409525Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0409726Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0410221Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0410429Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0410636Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0410641Z 
2025-07-17T11:07:27.0410731Z SIGABRT(6), PID: 1828346, Thread 1828543: 
2025-07-17T11:07:27.0411192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0411400Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0411671Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0411868Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0412363Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0412638Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0412845Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0412849Z 
2025-07-17T11:07:27.0412940Z SIGABRT(6), PID: 1828344, Thread 1828492: 
2025-07-17T11:07:27.0413470Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0413677Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0413882Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0414074Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0414566Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0414772Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0414981Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0414985Z 
2025-07-17T11:07:27.0415074Z SIGABRT(6), PID: 1828346, Thread 1828545: 
2025-07-17T11:07:27.0415539Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0415743Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0416005Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0416199Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0416770Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0416979Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0417186Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0417191Z 
2025-07-17T11:07:27.0417278Z SIGABRT(6), PID: 1828344, Thread 1828494: 
2025-07-17T11:07:27.0417754Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0417959Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0418160Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0418354Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0418854Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0419055Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0419262Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0419267Z 
2025-07-17T11:07:27.0419354Z SIGABRT(6), PID: 1828346, Thread 1828547: 
2025-07-17T11:07:27.0419903Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0420106Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0420307Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0420574Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0421067Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0421269Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0421547Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0421554Z 
2025-07-17T11:07:27.0421647Z SIGABRT(6), PID: 1828344, Thread 1828496: 
2025-07-17T11:07:27.0422114Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0422315Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0422517Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0422713Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0423201Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0423404Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0423614Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0423619Z 
2025-07-17T11:07:27.0423705Z SIGABRT(6), PID: 1828342, Thread 1833242: 
2025-07-17T11:07:27.0424167Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0424372Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0424642Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0424841Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0425213Z frame #4: <unknown function> + 0x5a02023 (0x7fed1699f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0425783Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed169a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0426376Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed1699a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0426755Z frame #7: <unknown function> + 0x79d655 (0x7fed27e8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0426988Z frame #8: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0427195Z frame #9: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0427407Z frame #10: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0427411Z 
2025-07-17T11:07:27.0427504Z SIGABRT(6), PID: 1828346, Thread 1828549: 
2025-07-17T11:07:27.0427965Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0428238Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0428438Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0428633Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0429189Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0429394Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0429600Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0429664Z 
2025-07-17T11:07:27.0429758Z SIGABRT(6), PID: 1828344, Thread 1828498: 
2025-07-17T11:07:27.0430221Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0430426Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0430628Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0430826Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0431317Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0431522Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0431729Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0431735Z 
2025-07-17T11:07:27.0431824Z SIGABRT(6), PID: 1828346, Thread 1828551: 
2025-07-17T11:07:27.0432282Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0432488Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0432687Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0432971Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0433463Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0433671Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0433878Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0433887Z 
2025-07-17T11:07:27.0433975Z SIGABRT(6), PID: 1828344, Thread 1828500: 
2025-07-17T11:07:27.0434440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0434641Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0434847Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0435040Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0435531Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0435743Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0436020Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0436024Z 
2025-07-17T11:07:27.0436111Z SIGABRT(6), PID: 1828346, Thread 1828553: 
2025-07-17T11:07:27.0436574Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0436842Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0437045Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0437236Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0437789Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0437998Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0438206Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0438210Z 
2025-07-17T11:07:27.0438297Z SIGABRT(6), PID: 1828344, Thread 1828502: 
2025-07-17T11:07:27.0438763Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0438968Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0439168Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0439359Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0439852Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0440054Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0440265Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0440270Z 
2025-07-17T11:07:27.0440355Z SIGABRT(6), PID: 1828346, Thread 1828555: 
2025-07-17T11:07:27.0440881Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0441090Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0441290Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0441485Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0441976Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0442179Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0442385Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0442389Z 
2025-07-17T11:07:27.0442476Z SIGABRT(6), PID: 1828344, Thread 1828504: 
2025-07-17T11:07:27.0442945Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0443149Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0443351Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0443546Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0444106Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0444309Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0444515Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0444583Z 
2025-07-17T11:07:27.0444674Z SIGABRT(6), PID: 1828346, Thread 1828557: 
2025-07-17T11:07:27.0445138Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0445346Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0445602Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0445801Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0446290Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0446495Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0446698Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0446705Z 
2025-07-17T11:07:27.0446793Z SIGABRT(6), PID: 1828344, Thread 1828506: 
2025-07-17T11:07:27.0447251Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0447456Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0447657Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0447855Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0448345Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0448549Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0448818Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0448825Z 
2025-07-17T11:07:27.0448917Z SIGABRT(6), PID: 1828342, Thread 1833243: 
2025-07-17T11:07:27.0449378Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0449587Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0449789Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0449987Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0450356Z frame #4: <unknown function> + 0x5a02023 (0x7fed1699f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0450922Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed169a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0451517Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed1699a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0451897Z frame #7: <unknown function> + 0x79d655 (0x7fed27e8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0452198Z frame #8: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0452408Z frame #9: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0452622Z frame #10: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0452626Z 
2025-07-17T11:07:27.0452780Z SIGABRT(6), PID: 1828346, Thread 1828559: 
2025-07-17T11:07:27.0453244Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0453450Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0453648Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0453905Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0454403Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0454608Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0454812Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0454821Z 
2025-07-17T11:07:27.0454909Z SIGABRT(6), PID: 1828344, Thread 1828507: 
2025-07-17T11:07:27.0455372Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0455574Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0455776Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0456029Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0456525Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0456726Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0457016Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0457021Z 
2025-07-17T11:07:27.0457113Z SIGABRT(6), PID: 1828346, Thread 1828562: 
2025-07-17T11:07:27.0457577Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0457782Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0457984Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0458178Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0458668Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0458869Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0459080Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0459084Z 
2025-07-17T11:07:27.0459170Z SIGABRT(6), PID: 1828344, Thread 1828508: 
2025-07-17T11:07:27.0459633Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0459835Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0460119Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0460313Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0460802Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0461084Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0461291Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0461295Z 
2025-07-17T11:07:27.0461381Z SIGABRT(6), PID: 1828346, Thread 1828565: 
2025-07-17T11:07:27.0461938Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0462145Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0462349Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0462543Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0463034Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0463239Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0463448Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0463453Z 
2025-07-17T11:07:27.0463539Z SIGABRT(6), PID: 1828344, Thread 1828509: 
2025-07-17T11:07:27.0464004Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0464206Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0464408Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0464599Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0465154Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0465362Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0465566Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0465569Z 
2025-07-17T11:07:27.0465660Z SIGABRT(6), PID: 1828346, Thread 1828568: 
2025-07-17T11:07:27.0466115Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0466321Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0466520Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0466714Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0467207Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0467410Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0467613Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0467619Z 
2025-07-17T11:07:27.0467708Z SIGABRT(6), PID: 1828344, Thread 1828510: 
2025-07-17T11:07:27.0468350Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0468555Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0468755Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0469017Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0469506Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0469713Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0469974Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0469981Z 
2025-07-17T11:07:27.0470071Z SIGABRT(6), PID: 1828344, Thread 1828511: 
2025-07-17T11:07:27.0470535Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0470737Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0470939Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0471135Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0471624Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0471829Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0472033Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0472038Z 
2025-07-17T11:07:27.0472127Z SIGABRT(6), PID: 1828346, Thread 1828571: 
2025-07-17T11:07:27.0472586Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0472792Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0473054Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0473252Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0473742Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0473949Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0474154Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0474158Z 
2025-07-17T11:07:27.0474248Z SIGABRT(6), PID: 1828344, Thread 1828512: 
2025-07-17T11:07:27.0474705Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0474914Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0475112Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0475308Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0475801Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0476071Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0476280Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0476284Z 
2025-07-17T11:07:27.0476371Z SIGABRT(6), PID: 1828346, Thread 1828574: 
2025-07-17T11:07:27.0476850Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0477121Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0477322Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0477516Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0478076Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0478286Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0478495Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0478499Z 
2025-07-17T11:07:27.0478587Z SIGABRT(6), PID: 1828342, Thread 1833244: 
2025-07-17T11:07:27.0479056Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0479260Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0479461Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0479655Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0480027Z frame #4: <unknown function> + 0x5a02023 (0x7fed1699f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0480597Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed169a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0481263Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed1699a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0481651Z frame #7: <unknown function> + 0x79d655 (0x7fed27e8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0481889Z frame #8: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0482098Z frame #9: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0482315Z frame #10: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0482321Z 
2025-07-17T11:07:27.0482411Z SIGABRT(6), PID: 1828344, Thread 1828513: 
2025-07-17T11:07:27.0482882Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0483086Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0483293Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0483490Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0483988Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0484194Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0484472Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0484476Z 
2025-07-17T11:07:27.0484566Z SIGABRT(6), PID: 1828346, Thread 1828577: 
2025-07-17T11:07:27.0485030Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0485302Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0485503Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0485701Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0486250Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0486464Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0486668Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0486672Z 
2025-07-17T11:07:27.0486761Z SIGABRT(6), PID: 1828344, Thread 1828514: 
2025-07-17T11:07:27.0487223Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0487429Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0487628Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0487823Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0488317Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0488528Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0488731Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0488735Z 
2025-07-17T11:07:27.0488824Z SIGABRT(6), PID: 1828346, Thread 1828580: 
2025-07-17T11:07:27.0489350Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0489558Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0489757Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0489957Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0490445Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0490658Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0490864Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0490868Z 
2025-07-17T11:07:27.0490960Z SIGABRT(6), PID: 1828344, Thread 1828515: 
2025-07-17T11:07:27.0491420Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0491624Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0491823Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0492020Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0492578Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0492787Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0492991Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0493074Z 
2025-07-17T11:07:27.0493168Z SIGABRT(6), PID: 1828346, Thread 1828583: 
2025-07-17T11:07:27.0493627Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0493836Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0494099Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0494305Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0494801Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0495009Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0495217Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0495225Z 
2025-07-17T11:07:27.0495312Z SIGABRT(6), PID: 1828344, Thread 1828516: 
2025-07-17T11:07:27.0495784Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0496048Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0496252Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0496450Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0496948Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0497153Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0497442Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0497447Z 
2025-07-17T11:07:27.0497538Z SIGABRT(6), PID: 1828346, Thread 1828585: 
2025-07-17T11:07:27.0498014Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0498216Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0498424Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0498617Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0499110Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0499317Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0499524Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0499529Z 
2025-07-17T11:07:27.0499615Z SIGABRT(6), PID: 1828344, Thread 1828517: 
2025-07-17T11:07:27.0500086Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0500375Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0500582Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0500776Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0501277Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0501559Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0501770Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0501775Z 
2025-07-17T11:07:27.0501862Z SIGABRT(6), PID: 1828346, Thread 1828588: 
2025-07-17T11:07:27.0502400Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0502614Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0502816Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0503011Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0503529Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0503737Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0503946Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0503952Z 
2025-07-17T11:07:27.0504039Z SIGABRT(6), PID: 1828344, Thread 1828518: 
2025-07-17T11:07:27.0504515Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0504720Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0504924Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0505119Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0505682Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0505892Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0506101Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0506106Z 
2025-07-17T11:07:27.0506194Z SIGABRT(6), PID: 1828346, Thread 1828591: 
2025-07-17T11:07:27.0506658Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0506863Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0507062Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0507260Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0507746Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0507953Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0508159Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0508227Z 
2025-07-17T11:07:27.0508321Z SIGABRT(6), PID: 1828342, Thread 1833247: 
2025-07-17T11:07:27.0508788Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0508993Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0509259Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0509460Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0509838Z frame #4: <unknown function> + 0x5a02023 (0x7fed1699f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0510469Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed169a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0511068Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed1699a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0511453Z frame #7: <unknown function> + 0x79d655 (0x7fed27e8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0511687Z frame #8: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0511896Z frame #9: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0512108Z frame #10: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0512112Z 
2025-07-17T11:07:27.0512202Z SIGABRT(6), PID: 1828344, Thread 1828519: 
2025-07-17T11:07:27.0512667Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0512876Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0513075Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0513269Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0513830Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0514041Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0514246Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0514339Z SIGABRT(6), PID: 1828346, Thread 1828595: 
2025-07-17T11:07:27.0514802Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0515008Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0515206Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0515401Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0515894Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0516098Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0516303Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0516310Z 
2025-07-17T11:07:27.0516317Z 
2025-07-17T11:07:27.0516403Z SIGABRT(6), PID: 1828344, Thread 1828520: 
2025-07-17T11:07:27.0516938Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0517139Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0517341Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0517600Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0518092Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0518294Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0518560Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0518654Z SIGABRT(6), PID: 1828346, Thread 1828599: 
2025-07-17T11:07:27.0519125Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0519328Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0519527Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0519723Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0520217Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0520423Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0520630Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0520635Z 
2025-07-17T11:07:27.0520639Z 
2025-07-17T11:07:27.0520724Z SIGABRT(6), PID: 1828346, Thread 1828603: 
2025-07-17T11:07:27.0521191Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0521391Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0521656Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0521852Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0522346Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0522547Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0522756Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0522760Z 
2025-07-17T11:07:27.0522846Z SIGABRT(6), PID: 1828344, Thread 1828521: 
2025-07-17T11:07:27.0523311Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0523516Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0523719Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0523913Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0524407Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0524697Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0524905Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0524909Z 
2025-07-17T11:07:27.0524997Z SIGABRT(6), PID: 1828346, Thread 1828607: 
2025-07-17T11:07:27.0525464Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0525729Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0525938Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0526131Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0526683Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0526890Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0527098Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0527102Z 
2025-07-17T11:07:27.0527190Z SIGABRT(6), PID: 1828344, Thread 1828522: 
2025-07-17T11:07:27.0527660Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0527868Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0528069Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0528269Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0528757Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0528964Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0529166Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0529170Z 
2025-07-17T11:07:27.0529260Z SIGABRT(6), PID: 1828346, Thread 1828610: 
2025-07-17T11:07:27.0529792Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0529998Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0530199Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0530398Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0530883Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0531089Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0531292Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0531298Z 
2025-07-17T11:07:27.0531389Z SIGABRT(6), PID: 1828344, Thread 1828523: 
2025-07-17T11:07:27.0531853Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0532058Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0532257Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0532525Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0533014Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0533221Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0533497Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0533502Z 
2025-07-17T11:07:27.0533594Z SIGABRT(6), PID: 1828346, Thread 1828613: 
2025-07-17T11:07:27.0534057Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0534321Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0534523Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0534721Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0535207Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0535411Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0535620Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0535624Z 
2025-07-17T11:07:27.0535713Z SIGABRT(6), PID: 1828344, Thread 1828524: 
2025-07-17T11:07:27.0536240Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0536447Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0536648Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0536844Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0537333Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0537622Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0537835Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0537844Z 
2025-07-17T11:07:27.0537932Z SIGABRT(6), PID: 1828346, Thread 1828617: 
2025-07-17T11:07:27.0538397Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0538599Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0538802Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0538997Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0539488Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0539694Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0539899Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0539903Z 
2025-07-17T11:07:27.0539990Z SIGABRT(6), PID: 1828344, Thread 1828525: 
2025-07-17T11:07:27.0540455Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0540738Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0540941Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0541136Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0541710Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0541916Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0542124Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0542214Z SIGABRT(6), PID: 1828342, Thread 1833249: 
2025-07-17T11:07:27.0542751Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0542960Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0543168Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0543363Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0543740Z frame #4: <unknown function> + 0x5a02023 (0x7fed1699f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0544308Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed169a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0544902Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed1699a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0545284Z frame #7: <unknown function> + 0x79d655 (0x7fed27e8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0545519Z frame #8: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0545724Z frame #9: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0546002Z frame #10: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0546007Z 
2025-07-17T11:07:27.0546010Z 
2025-07-17T11:07:27.0546101Z SIGABRT(6), PID: 1828346, Thread 1828621: 
2025-07-17T11:07:27.0546573Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0546776Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0546979Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0547176Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0547674Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0547883Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0548089Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0548093Z 
2025-07-17T11:07:27.0548184Z SIGABRT(6), PID: 1828344, Thread 1828526: 
2025-07-17T11:07:27.0548649Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0548924Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0549125Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0549320Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0549813Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0550088Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0550293Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0550298Z 
2025-07-17T11:07:27.0550388Z SIGABRT(6), PID: 1828346, Thread 1828624: 
2025-07-17T11:07:27.0550918Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0551127Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0551326Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0551522Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0552013Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0552218Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0552423Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0552428Z 
2025-07-17T11:07:27.0552519Z SIGABRT(6), PID: 1828344, Thread 1828527: 
2025-07-17T11:07:27.0552981Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0553185Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0553385Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0553582Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0554156Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0554366Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0554569Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0554574Z 
2025-07-17T11:07:27.0554663Z SIGABRT(6), PID: 1828344, Thread 1828528: 
2025-07-17T11:07:27.0555127Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0555332Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0555530Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0555731Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0556220Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0556427Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0556633Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0556699Z 
2025-07-17T11:07:27.0556791Z SIGABRT(6), PID: 1828346, Thread 1828628: 
2025-07-17T11:07:27.0557258Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0557464Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0557737Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0557930Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0558427Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0558692Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0558907Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0558914Z 
2025-07-17T11:07:27.0559002Z SIGABRT(6), PID: 1828344, Thread 1828529: 
2025-07-17T11:07:27.0559471Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0559676Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0559879Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0560072Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0560564Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0560768Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0560977Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0560981Z 
2025-07-17T11:07:27.0561068Z SIGABRT(6), PID: 1828346, Thread 1828632: 
2025-07-17T11:07:27.0561533Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0561803Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0562008Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0562200Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0562692Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0562896Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0563101Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0563104Z 
2025-07-17T11:07:27.0563191Z SIGABRT(6), PID: 1828344, Thread 1828530: 
2025-07-17T11:07:27.0563659Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0563860Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0564061Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0564253Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0564745Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0565022Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0565232Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0565236Z 
2025-07-17T11:07:27.0565322Z SIGABRT(6), PID: 1828346, Thread 1828636: 
2025-07-17T11:07:27.0565847Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0566051Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0566254Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0566507Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0567002Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0567207Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0567415Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0567419Z 
2025-07-17T11:07:27.0567508Z SIGABRT(6), PID: 1828344, Thread 1828531: 
2025-07-17T11:07:27.0567977Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0568182Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0568382Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0568579Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0569067Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0569273Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0569476Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0569481Z 
2025-07-17T11:07:27.0569643Z SIGABRT(6), PID: 1828346, Thread 1828641: 
2025-07-17T11:07:27.0570106Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0570311Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0570514Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0570711Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0571203Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0571409Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0571616Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0571620Z 
2025-07-17T11:07:27.0571709Z SIGABRT(6), PID: 1828342, Thread 1833252: 
2025-07-17T11:07:27.0572167Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7fece0c68b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0572375Z frame #1: <unknown function> + 0x42520 (0x7fedd2521520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0572574Z frame #2: <unknown function> + 0x91117 (0x7fedd2570117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0572843Z frame #3: pthread_cond_wait + 0x211 (0x7fedd2572a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0573215Z frame #4: <unknown function> + 0x5a02023 (0x7fed1699f023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0573784Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7fed169a31ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0574437Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7fed1699a1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0574877Z frame #7: <unknown function> + 0x79d655 (0x7fed27e8a655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0575116Z frame #8: <unknown function> + 0xdc253 (0x7fecdeed5253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0575327Z frame #9: <unknown function> + 0x94ac3 (0x7fedd2573ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0575536Z frame #10: <unknown function> + 0x126850 (0x7fedd2605850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0575541Z 
2025-07-17T11:07:27.0575632Z SIGABRT(6), PID: 1828344, Thread 1828532: 
2025-07-17T11:07:27.0576175Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0576385Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0576585Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0576788Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0577282Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0577490Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0577695Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0577706Z 
2025-07-17T11:07:27.0577791Z SIGABRT(6), PID: 1828346, Thread 1828645: 
2025-07-17T11:07:27.0578334Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0578540Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0578749Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0578944Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0579439Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0579642Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0579846Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0579852Z 
2025-07-17T11:07:27.0579940Z SIGABRT(6), PID: 1828346, Thread 1828649: 
2025-07-17T11:07:27.0580400Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0580601Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0580805Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0581080Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0581572Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0581774Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0582064Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0582069Z 
2025-07-17T11:07:27.0582158Z SIGABRT(6), PID: 1828344, Thread 1828533: 
2025-07-17T11:07:27.0582625Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0582920Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0583128Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0583321Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0583816Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0584021Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0584226Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0584230Z 
2025-07-17T11:07:27.0584317Z SIGABRT(6), PID: 1828346, Thread 1828653: 
2025-07-17T11:07:27.0584778Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0584981Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0585185Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0585377Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0585871Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0586140Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0586350Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0586354Z 
2025-07-17T11:07:27.0586440Z SIGABRT(6), PID: 1828344, Thread 1828534: 
2025-07-17T11:07:27.0586907Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0587112Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0587315Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0587507Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0588008Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0588214Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0588423Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0588427Z 
2025-07-17T11:07:27.0588513Z SIGABRT(6), PID: 1828346, Thread 1828657: 
2025-07-17T11:07:27.0588979Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0589252Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0589452Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0589648Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0590197Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0590407Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0590610Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0590614Z 
2025-07-17T11:07:27.0590761Z SIGABRT(6), PID: 1828344, Thread 1828535: 
2025-07-17T11:07:27.0591225Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0591430Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0591629Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0591828Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0592319Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0592525Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0592730Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0592734Z 
2025-07-17T11:07:27.0592822Z SIGABRT(6), PID: 1828346, Thread 1828661: 
2025-07-17T11:07:27.0593282Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0593486Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0593685Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0594050Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0594540Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0594750Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0594956Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0594963Z 
2025-07-17T11:07:27.0595055Z SIGABRT(6), PID: 1828344, Thread 1828536: 
2025-07-17T11:07:27.0595518Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0595722Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0595924Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0596119Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0596609Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0596816Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0597091Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0597095Z 
2025-07-17T11:07:27.0597184Z SIGABRT(6), PID: 1828346, Thread 1828665: 
2025-07-17T11:07:27.0597644Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0597849Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0598114Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0598311Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0598853Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0599062Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0599269Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0599276Z 
2025-07-17T11:07:27.0599363Z SIGABRT(6), PID: 1828344, Thread 1828537: 
2025-07-17T11:07:27.0599835Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0600040Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0600245Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0600438Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0600930Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0601133Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0601340Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0601343Z 
2025-07-17T11:07:27.0601431Z SIGABRT(6), PID: 1828346, Thread 1828669: 
2025-07-17T11:07:27.0601961Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0602170Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0602374Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0602566Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0603063Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0603268Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0603473Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0603477Z 
2025-07-17T11:07:27.0603563Z SIGABRT(6), PID: 1828344, Thread 1828538: 
2025-07-17T11:07:27.0604034Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0604236Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0604437Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0604631Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0605125Z frame #4: <unknown function> + 0x3475ab (0x7f9f4822d5ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0605400Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0605609Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0605613Z 
2025-07-17T11:07:27.0605764Z SIGABRT(6), PID: 1828344, Thread 1829404: 
2025-07-17T11:07:27.0606234Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0606679Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f9e59a8a08a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0606952Z frame #2: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0607147Z frame #3: pthread_kill + 0x12c (0x7f9f4b3979fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0607314Z frame #4: raise + 0x16 (0x7f9f4b343476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0607471Z frame #5: abort + 0xd3 (0x7f9f4b3297f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0607710Z frame #6: <unknown function> + 0x972cc (0x7f9dc76132cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0607942Z frame #7: <unknown function> + 0x95f47 (0x7f9dc7611f47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0608169Z frame #8: <unknown function> + 0x34b71 (0x7f9dc75b0b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0608373Z frame #9: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0608585Z frame #10: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0608591Z 
2025-07-17T11:07:27.0608679Z SIGABRT(6), PID: 1828346, Thread 1828674: 
2025-07-17T11:07:27.0609150Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0609355Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0609559Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0609821Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0610316Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0610520Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0610728Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0610732Z 
2025-07-17T11:07:27.0610822Z SIGABRT(6), PID: 1828346, Thread 1828679: 
2025-07-17T11:07:27.0611283Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0611488Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0611690Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0611889Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0612379Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0612586Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0612789Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0612856Z 
2025-07-17T11:07:27.0612948Z SIGABRT(6), PID: 1828344, Thread 1832328: 
2025-07-17T11:07:27.0613408Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0613618Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0613874Z frame #2: __poll + 0x4f (0x7f9f4b419bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0614284Z frame #3: <unknown function> + 0x489f (0x7f9f4af2589f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:27.0614508Z frame #4: <unknown function> + 0x1483bc (0x555cf89d13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0614815Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0615045Z frame #6: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0615279Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0615499Z frame #8: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0615745Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x555cf89b9a65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0616033Z frame #10: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0616283Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x555cf89b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0616510Z frame #12: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0616748Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0616971Z frame #14: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0617206Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0617431Z frame #16: <unknown function> + 0x147734 (0x555cf89d0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0617668Z frame #17: <unknown function> + 0x21ca82 (0x555cf8aa5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0617967Z frame #18: <unknown function> + 0x21c9e4 (0x555cf8aa59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0618187Z frame #19: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0618397Z frame #20: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0618401Z 
2025-07-17T11:07:27.0618494Z SIGABRT(6), PID: 1828346, Thread 1828683: 
2025-07-17T11:07:27.0618961Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0619174Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0619375Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0619575Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0620077Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0620285Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0620491Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0620497Z 
2025-07-17T11:07:27.0620589Z SIGABRT(6), PID: 1828346, Thread 1828686: 
2025-07-17T11:07:27.0621131Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0621339Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0621537Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0621814Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0622310Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0622515Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0622793Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0622798Z 
2025-07-17T11:07:27.0622892Z SIGABRT(6), PID: 1828346, Thread 1828690: 
2025-07-17T11:07:27.0623360Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0623567Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0623766Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0623966Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0624461Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0624668Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0624870Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0624962Z SIGABRT(6), PID: 1828344, Thread 1832331: 
2025-07-17T11:07:27.0625425Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0625635Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0625900Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0626126Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f9f4b3952dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0626567Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f9e857d2048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0626992Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f9e857d4bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0627226Z frame #6: <unknown function> + 0xdc253 (0x7f9e57cf6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0627430Z frame #7: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0627637Z frame #8: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0627642Z 
2025-07-17T11:07:27.0627647Z 
2025-07-17T11:07:27.0627734Z SIGABRT(6), PID: 1828346, Thread 1828694: 
2025-07-17T11:07:27.0628204Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0628407Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0628610Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0628804Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0629368Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0629571Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0629777Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0629841Z 
2025-07-17T11:07:27.0629933Z SIGABRT(6), PID: 1828346, Thread 1828697: 
2025-07-17T11:07:27.0630398Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0630602Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0630867Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0631066Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0631559Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0631759Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0631969Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0631974Z 
2025-07-17T11:07:27.0632063Z SIGABRT(6), PID: 1828344, Thread 1832332: 
2025-07-17T11:07:27.0632533Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0632736Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0632942Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0633161Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f9f4b3952dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0633627Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f9e857cc741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0633856Z frame #5: <unknown function> + 0xdc253 (0x7f9e57cf6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0634126Z frame #6: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0634336Z frame #7: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0634341Z 
2025-07-17T11:07:27.0634432Z SIGABRT(6), PID: 1828344, Thread 1832408: 
2025-07-17T11:07:27.0634898Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0635105Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0635269Z frame #2: ioctl + 0x3f (0x7f9f4b41b94f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0635504Z frame #3: <unknown function> + 0x13eef0 (0x7f9dc76baef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0635731Z frame #4: <unknown function> + 0x1378c4 (0x7f9dc76b38c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0635966Z frame #5: <unknown function> + 0x95b2b (0x7f9dc7611b2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0636189Z frame #6: <unknown function> + 0x34b71 (0x7f9dc75b0b71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0636395Z frame #7: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0636605Z frame #8: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0636676Z 
2025-07-17T11:07:27.0636770Z SIGABRT(6), PID: 1828346, Thread 1828702: 
2025-07-17T11:07:27.0637235Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0637442Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0637643Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0637907Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0638403Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0638666Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0638876Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0638884Z 
2025-07-17T11:07:27.0638973Z SIGABRT(6), PID: 1828346, Thread 1828706: 
2025-07-17T11:07:27.0639440Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0639640Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0639846Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0640039Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0640537Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0640740Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0640948Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0640953Z 
2025-07-17T11:07:27.0641039Z SIGABRT(6), PID: 1828344, Thread 1833156: 
2025-07-17T11:07:27.0641506Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0641773Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0641947Z frame #2: __poll + 0x4f (0x7f9f4b419bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0642158Z frame #3: <unknown function> + 0xd4540 (0x7f9e0f82a540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0642366Z frame #4: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0642571Z frame #5: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0642577Z 
2025-07-17T11:07:27.0642667Z SIGABRT(6), PID: 1828346, Thread 1828710: 
2025-07-17T11:07:27.0643125Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0643330Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0643530Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0643727Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0644224Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0644429Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0644633Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0644724Z 
2025-07-17T11:07:27.0644817Z SIGABRT(6), PID: 1828344, Thread 1833163: 
2025-07-17T11:07:27.0645280Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0645483Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0645714Z frame #2: __poll + 0x4f (0x7f9f4b419bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0645927Z frame #3: <unknown function> + 0xd65a2 (0x7f9e0f82c5a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0646148Z frame #4: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0646432Z frame #5: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0646436Z 
2025-07-17T11:07:27.0646538Z SIGABRT(6), PID: 1828344, Thread 1833168: 
2025-07-17T11:07:27.0647019Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0647228Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0647435Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0647634Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0647851Z frame #4: <unknown function> + 0xd1e7f (0x7f9e0f827e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0648056Z frame #5: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0648265Z frame #6: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0648269Z 
2025-07-17T11:07:27.0648359Z SIGABRT(6), PID: 1828346, Thread 1828714: 
2025-07-17T11:07:27.0648828Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0649031Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0649235Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0649495Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0649998Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0650205Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0650411Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0650417Z 
2025-07-17T11:07:27.0650508Z SIGABRT(6), PID: 1828346, Thread 1828718: 
2025-07-17T11:07:27.0650976Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0651181Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0651386Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0651585Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0652080Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0652288Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0652494Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0652654Z SIGABRT(6), PID: 1828344, Thread 1833182: 
2025-07-17T11:07:27.0653120Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0653327Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0653599Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0653811Z frame #3: <unknown function> + 0x9ca53 (0x7f9f4b39da53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0654065Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x555cf899dbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0654357Z frame #5: <unknown function> + 0x1d314f (0x555cf8a5c14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0654586Z frame #6: <unknown function> + 0x1d2f6b (0x555cf8a5bf6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0654812Z frame #7: <unknown function> + 0x6df36 (0x555cf88f6f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0655050Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0655281Z frame #9: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0655526Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0655757Z frame #11: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0656052Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0656279Z frame #13: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0656513Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0656736Z frame #15: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0656966Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0657190Z frame #17: <unknown function> + 0x147734 (0x555cf89d0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0657491Z frame #18: <unknown function> + 0x21ca82 (0x555cf8aa5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0657716Z frame #19: <unknown function> + 0x21c9e4 (0x555cf8aa59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0657927Z frame #20: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0658139Z frame #21: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0658146Z 
2025-07-17T11:07:27.0658149Z 
2025-07-17T11:07:27.0658238Z SIGABRT(6), PID: 1828346, Thread 1828722: 
2025-07-17T11:07:27.0658717Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0658925Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0659130Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0659330Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0659830Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0660034Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0660243Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0660326Z 
2025-07-17T11:07:27.0660419Z SIGABRT(6), PID: 1828344, Thread 1833196: 
2025-07-17T11:07:27.0660893Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0661096Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0661338Z frame #2: read + 0x4c (0x7f9f4b41581c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0661521Z frame #3: _Py_read + 0x4c (0x555cf8a63a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0661748Z frame #4: <unknown function> + 0x208c3b (0x555cf8a91c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0661970Z frame #5: <unknown function> + 0x14648d (0x555cf89cf48d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0662286Z frame #6: PyObject_VectorcallMethod + 0x85 (0x555cf89de215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0662508Z frame #7: <unknown function> + 0x208ae7 (0x555cf8a91ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0662726Z frame #8: <unknown function> + 0x208472 (0x555cf8a91472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0662943Z frame #9: <unknown function> + 0x1da6ee (0x555cf8a636ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0663165Z frame #10: <unknown function> + 0x1483bc (0x555cf89d13bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0663407Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0663634Z frame #12: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0663868Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x555cf89b53bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0664089Z frame #14: <unknown function> + 0x147734 (0x555cf89d0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0664336Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x555cf89b7c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0664563Z frame #16: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0664796Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0665019Z frame #18: _PyFunction_Vectorcall + 0x6c (0x555cf89c50ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0665313Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x555cf89b57b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0665538Z frame #20: <unknown function> + 0x147734 (0x555cf89d0734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0665756Z frame #21: <unknown function> + 0x21ca82 (0x555cf8aa5a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0665978Z frame #22: <unknown function> + 0x21c9e4 (0x555cf8aa59e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0666187Z frame #23: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0666413Z frame #24: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0666417Z 
2025-07-17T11:07:27.0666511Z SIGABRT(6), PID: 1828346, Thread 1828726: 
2025-07-17T11:07:27.0666989Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0667203Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0667412Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0667612Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0668123Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0668402Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0668613Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0668618Z 
2025-07-17T11:07:27.0668707Z SIGABRT(6), PID: 1828346, Thread 1828730: 
2025-07-17T11:07:27.0669182Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0669451Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0669656Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0669853Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0670415Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0670625Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0670836Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0670842Z 
2025-07-17T11:07:27.0670931Z SIGABRT(6), PID: 1828346, Thread 1828734: 
2025-07-17T11:07:27.0671401Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0671609Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0671810Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0672008Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0672499Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0672709Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0672913Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0672919Z 
2025-07-17T11:07:27.0673012Z SIGABRT(6), PID: 1828346, Thread 1828737: 
2025-07-17T11:07:27.0673565Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0673773Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0673976Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0674175Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0674668Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0674876Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0675080Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0675086Z 
2025-07-17T11:07:27.0675178Z SIGABRT(6), PID: 1828346, Thread 1828743: 
2025-07-17T11:07:27.0675640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0675846Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0676049Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0676316Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0676809Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0677013Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0677282Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0677286Z 
2025-07-17T11:07:27.0677381Z SIGABRT(6), PID: 1828346, Thread 1828747: 
2025-07-17T11:07:27.0677842Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0678105Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0678311Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0678513Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0679003Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0679212Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0679418Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0679423Z 
2025-07-17T11:07:27.0679513Z SIGABRT(6), PID: 1828346, Thread 1828751: 
2025-07-17T11:07:27.0679978Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0680183Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0680383Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0680581Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0681070Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0681343Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0681553Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0681560Z 
2025-07-17T11:07:27.0681648Z SIGABRT(6), PID: 1828344, Thread 1833232: 
2025-07-17T11:07:27.0682122Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0682330Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0682531Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0682726Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0683111Z frame #4: <unknown function> + 0x5a02023 (0x7f9e8f7c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0683687Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9e8f7c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0684285Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9e8f7bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0684736Z frame #7: <unknown function> + 0x79d655 (0x7f9ea0ca7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0684972Z frame #8: <unknown function> + 0xdc253 (0x7f9e57cf6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0685181Z frame #9: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0685396Z frame #10: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0685468Z 
2025-07-17T11:07:27.0685558Z SIGABRT(6), PID: 1828347, Thread 1828347: 
2025-07-17T11:07:27.0686035Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0686243Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0686537Z frame #2: <unknown function> + 0x7741f (0x7f195b5ce41f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0686772Z frame #3: <unknown function> + 0x76c8e (0x7f195b5cdc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0687004Z frame #4: <unknown function> + 0x6ad69 (0x7f195b5c1d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0687220Z frame #5: <unknown function> + 0x3f54db (0x7f19ec2264db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0687435Z frame #6: <unknown function> + 0x3e11d8 (0x7f19ec2121d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0687646Z frame #7: <unknown function> + 0xb2f86 (0x7f19ebee3f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0687857Z frame #8: <unknown function> + 0x8e501 (0x7f19ebebf501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0688241Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f1a16cc67ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:27.0688625Z frame #10: <unknown function> + 0xd307d9 (0x7f1a352197d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0688851Z frame #11: <unknown function> + 0x139902 (0x55f4794f4902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0689102Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55f4794eba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0689327Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0689633Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55f4794eba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0689859Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0690099Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0690321Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0690556Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0690773Z frame #19: <unknown function> + 0x1474b2 (0x55f4795024b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0690976Z frame #20: PyObject_Call + 0xbc (0x55f479502e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0691195Z frame #21: <unknown function> + 0x1ff69d (0x55f4795ba69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0691421Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55f4794f0023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0691660Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55f4794ebcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0691883Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0692117Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55f4794e73bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0692337Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0692635Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55f4794e73bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0692858Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0693099Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55f4794eba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0693424Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55f4794ef4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0693642Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55f479500ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0693860Z frame #32: <unknown function> + 0x205949 (0x55f4795c0949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0694077Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55f4794f0023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0694386Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55f4794ebcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0694614Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0694849Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55f4794e73bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0695068Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0695304Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55f4794e73bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0695525Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0695757Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55f4794e73bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0696030Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0696272Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55f4794e83f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0696495Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0696881Z frame #44: <unknown function> + 0x7ab3f7 (0x7f1a34c943f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0697100Z frame #45: <unknown function> + 0x13bc88 (0x55f4794f6c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0697393Z frame #46: PyObject_Call + 0x20f (0x55f479502f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0697659Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55f4794ec98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0697893Z frame #48: <unknown function> + 0x147608 (0x55f479502608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0698145Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f4794e9c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0698375Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0698614Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55f4794e73bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0698839Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0699079Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55f4794e83f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0699303Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0699534Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55f4794e73bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0699757Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0699994Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f4794e9c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0700218Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0700541Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f4794e9c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0700763Z frame #60: <unknown function> + 0x147608 (0x55f479502608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0700999Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f4794e9c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0701291Z frame #62: <unknown function> + 0x147608 (0x55f479502608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0701385Z SIGABRT(6), PID: 1828346, Thread 1828754: 
2025-07-17T11:07:27.0701873Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0702085Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0702393Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0702601Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0703111Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0703319Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0703534Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0703538Z 
2025-07-17T11:07:27.0703542Z 
2025-07-17T11:07:27.0703633Z SIGABRT(6), PID: 1828346, Thread 1828758: 
2025-07-17T11:07:27.0704105Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0704311Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0704519Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0704727Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0705249Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0705526Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0705752Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0705756Z 
2025-07-17T11:07:27.0705848Z SIGABRT(6), PID: 1828347, Thread 1828558: 
2025-07-17T11:07:27.0706334Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0706548Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0706760Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0706961Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0707467Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0707678Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0707887Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0707892Z 
2025-07-17T11:07:27.0707981Z SIGABRT(6), PID: 1828346, Thread 1828761: 
2025-07-17T11:07:27.0708455Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0708732Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0708935Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0709133Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0709693Z frame #4: <unknown function> + 0x3475ab (0x7f43481085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0709902Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0710107Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0710111Z 
2025-07-17T11:07:27.0710261Z SIGABRT(6), PID: 1828347, Thread 1828561: 
2025-07-17T11:07:27.0710736Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0710946Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0711150Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0711350Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0711850Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0712059Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0712266Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0712273Z 
2025-07-17T11:07:27.0712362Z SIGABRT(6), PID: 1828346, Thread 1829408: 
2025-07-17T11:07:27.0712833Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0713288Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f425996508a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0713493Z frame #2: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0713751Z frame #3: pthread_kill + 0x12c (0x7f434b2729fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0713928Z frame #4: raise + 0x16 (0x7f434b21e476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0714090Z frame #5: abort + 0xd3 (0x7f434b2047f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0714332Z frame #6: <unknown function> + 0x972cc (0x7f41c74ee2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0714562Z frame #7: <unknown function> + 0x95f47 (0x7f41c74ecf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0714785Z frame #8: <unknown function> + 0x34b71 (0x7f41c748bb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0714994Z frame #9: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0715206Z frame #10: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0715213Z 
2025-07-17T11:07:27.0715307Z SIGABRT(6), PID: 1828347, Thread 1828564: 
2025-07-17T11:07:27.0715778Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0715992Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0716194Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0716394Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0716984Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0717194Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0717399Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0717465Z 
2025-07-17T11:07:27.0717560Z SIGABRT(6), PID: 1828346, Thread 1833136: 
2025-07-17T11:07:27.0718031Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0718235Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0718584Z frame #2: __poll + 0x4f (0x7f434b2f4bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0718999Z frame #3: <unknown function> + 0x489f (0x7f434ae0089f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:27.0719225Z frame #4: <unknown function> + 0x1483bc (0x556877b543bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0719464Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0719693Z frame #6: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0719928Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0720150Z frame #8: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0720392Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x556877b3ca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0720621Z frame #10: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0720870Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x556877b3ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0721096Z frame #12: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0721334Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0721622Z frame #14: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0721864Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0722080Z frame #16: <unknown function> + 0x147734 (0x556877b53734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0722300Z frame #17: <unknown function> + 0x21ca82 (0x556877c28a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0722518Z frame #18: <unknown function> + 0x21c9e4 (0x556877c289e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0722734Z frame #19: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0722944Z frame #20: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0722951Z 
2025-07-17T11:07:27.0723042Z SIGABRT(6), PID: 1828347, Thread 1828567: 
2025-07-17T11:07:27.0723519Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0723738Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0723947Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0724146Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0724651Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0724929Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0725141Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0725145Z 
2025-07-17T11:07:27.0725234Z SIGABRT(6), PID: 1828347, Thread 1828570: 
2025-07-17T11:07:27.0725772Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0725979Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0726181Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0726436Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0733033Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0733309Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0733542Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0733550Z 
2025-07-17T11:07:27.0733650Z SIGABRT(6), PID: 1828344, Thread 1833233: 
2025-07-17T11:07:27.0734155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0734373Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0734581Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0734784Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0735167Z frame #4: <unknown function> + 0x5a02023 (0x7f9e8f7c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0735743Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9e8f7c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0736555Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9e8f7bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0736962Z frame #7: <unknown function> + 0x79d655 (0x7f9ea0ca7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0737212Z frame #8: <unknown function> + 0xdc253 (0x7f9e57cf6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0737426Z frame #9: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0737647Z frame #10: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0737652Z 
2025-07-17T11:07:27.0737745Z SIGABRT(6), PID: 1828346, Thread 1833137: 
2025-07-17T11:07:27.0738227Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0738444Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0738649Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0738870Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f434b2702dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0739314Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f42856ad048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0739822Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f42856afbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0740058Z frame #6: <unknown function> + 0xdc253 (0x7f4257bd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0740264Z frame #7: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0740579Z frame #8: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0740584Z 
2025-07-17T11:07:27.0740678Z SIGABRT(6), PID: 1828347, Thread 1828573: 
2025-07-17T11:07:27.0741150Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0741430Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0741638Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0741841Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0742349Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0742563Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0742773Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0742777Z 
2025-07-17T11:07:27.0742867Z SIGABRT(6), PID: 1828347, Thread 1828576: 
2025-07-17T11:07:27.0743335Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0743541Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0743745Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0743943Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0744437Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0744707Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0744920Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0744925Z 
2025-07-17T11:07:27.0745014Z SIGABRT(6), PID: 1828346, Thread 1833138: 
2025-07-17T11:07:27.0745478Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0745684Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0745882Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0746102Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f434b2702dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0746573Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f42856a7741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.0746803Z frame #5: <unknown function> + 0xdc253 (0x7f4257bd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0747006Z frame #6: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0747212Z frame #7: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0747218Z 
2025-07-17T11:07:27.0747304Z SIGABRT(6), PID: 1828346, Thread 1833141: 
2025-07-17T11:07:27.0747840Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0748042Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0748209Z frame #2: ioctl + 0x3f (0x7f434b2f694f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0748511Z frame #3: <unknown function> + 0x13eef0 (0x7f41c7595ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0748745Z frame #4: <unknown function> + 0x1378c4 (0x7f41c758e8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0748972Z frame #5: <unknown function> + 0x95b2b (0x7f41c74ecb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0749197Z frame #6: <unknown function> + 0x34b71 (0x7f41c748bb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0749471Z frame #7: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0749686Z frame #8: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0749690Z 
2025-07-17T11:07:27.0749777Z SIGABRT(6), PID: 1828347, Thread 1828578: 
2025-07-17T11:07:27.0750242Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0750448Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0750651Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0750848Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0751354Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0751561Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0751769Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0751774Z 
2025-07-17T11:07:27.0751864Z SIGABRT(6), PID: 1828347, Thread 1828581: 
2025-07-17T11:07:27.0752334Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0752609Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0752817Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0753017Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0753510Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0753718Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0753922Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0753926Z 
2025-07-17T11:07:27.0754016Z SIGABRT(6), PID: 1828346, Thread 1833154: 
2025-07-17T11:07:27.0754482Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0754686Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0754849Z frame #2: __poll + 0x4f (0x7f434b2f4bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0755061Z frame #3: <unknown function> + 0xd4540 (0x7f420f705540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0755262Z frame #4: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0755543Z frame #5: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0755549Z 
2025-07-17T11:07:27.0755637Z SIGABRT(6), PID: 1828346, Thread 1833161: 
2025-07-17T11:07:27.0756099Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0756366Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0756528Z frame #2: __poll + 0x4f (0x7f434b2f4bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0756738Z frame #3: <unknown function> + 0xd65a2 (0x7f420f7075a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0756944Z frame #4: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0757204Z frame #5: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0757300Z SIGABRT(6), PID: 1828347, Thread 1828584: 
2025-07-17T11:07:27.0757761Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0757969Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0758171Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0758370Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0758866Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0759073Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0759277Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0759285Z 
2025-07-17T11:07:27.0759288Z 
2025-07-17T11:07:27.0759380Z SIGABRT(6), PID: 1828346, Thread 1833173: 
2025-07-17T11:07:27.0759837Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0760041Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0760308Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0760509Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0760718Z frame #4: <unknown function> + 0xd1e7f (0x7f420f702e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.0760922Z frame #5: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0761129Z frame #6: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0761133Z 
2025-07-17T11:07:27.0761222Z SIGABRT(6), PID: 1828344, Thread 1833234: 
2025-07-17T11:07:27.0761685Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0761890Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0762091Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0762290Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0762664Z frame #4: <unknown function> + 0x5a02023 (0x7f9e8f7c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0763231Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9e8f7c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0763892Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9e8f7bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0764274Z frame #7: <unknown function> + 0x79d655 (0x7f9ea0ca7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0764569Z frame #8: <unknown function> + 0xdc253 (0x7f9e57cf6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0764772Z frame #9: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0764983Z frame #10: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0765045Z 
2025-07-17T11:07:27.0765136Z SIGABRT(6), PID: 1828347, Thread 1828587: 
2025-07-17T11:07:27.0765611Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0765816Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0766021Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0766222Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0766722Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0766925Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0767134Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0767140Z 
2025-07-17T11:07:27.0767228Z SIGABRT(6), PID: 1828346, Thread 1833177: 
2025-07-17T11:07:27.0767694Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0767895Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0768096Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0768366Z frame #3: <unknown function> + 0x9ca53 (0x7f434b278a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0768622Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x556877b20bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0768846Z frame #5: <unknown function> + 0x1d314f (0x556877bdf14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0769070Z frame #6: <unknown function> + 0x1d2f6b (0x556877bdef6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0769289Z frame #7: <unknown function> + 0x6df36 (0x556877a79f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0769532Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0769751Z frame #9: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0769991Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0770216Z frame #11: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0770450Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0770670Z frame #13: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0770902Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0771121Z frame #15: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0771449Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0771663Z frame #17: <unknown function> + 0x147734 (0x556877b53734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0771882Z frame #18: <unknown function> + 0x21ca82 (0x556877c28a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0772164Z frame #19: <unknown function> + 0x21c9e4 (0x556877c289e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0772373Z frame #20: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0772581Z frame #21: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0772586Z 
2025-07-17T11:07:27.0772677Z SIGABRT(6), PID: 1828347, Thread 1828590: 
2025-07-17T11:07:27.0773199Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0773417Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0773620Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0773820Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0774317Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0774525Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0774732Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0774736Z 
2025-07-17T11:07:27.0774827Z SIGABRT(6), PID: 1828346, Thread 1833187: 
2025-07-17T11:07:27.0775289Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0775493Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0775652Z frame #2: read + 0x4c (0x7f434b2f081c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0775831Z frame #3: _Py_read + 0x4c (0x556877be6a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0776206Z frame #4: <unknown function> + 0x208c3b (0x556877c14c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0776432Z frame #5: <unknown function> + 0x14648d (0x556877b5248d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0776673Z frame #6: PyObject_VectorcallMethod + 0x85 (0x556877b61215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0776894Z frame #7: <unknown function> + 0x208ae7 (0x556877c14ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0777107Z frame #8: <unknown function> + 0x208472 (0x556877c14472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0777326Z frame #9: <unknown function> + 0x1da6ee (0x556877be66ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0777546Z frame #10: <unknown function> + 0x1483bc (0x556877b543bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0777785Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0778011Z frame #12: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0778245Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x556877b383bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0778458Z frame #14: <unknown function> + 0x147734 (0x556877b53734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0778706Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x556877b3ac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0779010Z frame #16: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0779242Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0779461Z frame #18: _PyFunction_Vectorcall + 0x6c (0x556877b480ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0779690Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x556877b387b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0779979Z frame #20: <unknown function> + 0x147734 (0x556877b53734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0780197Z frame #21: <unknown function> + 0x21ca82 (0x556877c28a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0780412Z frame #22: <unknown function> + 0x21c9e4 (0x556877c289e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0780692Z frame #23: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0780905Z frame #24: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0780911Z 
2025-07-17T11:07:27.0781006Z SIGABRT(6), PID: 1828347, Thread 1828594: 
2025-07-17T11:07:27.0781469Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0781679Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0781891Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0782094Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0782592Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0782801Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0783009Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0783014Z 
2025-07-17T11:07:27.0783105Z SIGABRT(6), PID: 1828347, Thread 1828597: 
2025-07-17T11:07:27.0783566Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0783840Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0784044Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0784243Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0784736Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0784939Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0785149Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0785154Z 
2025-07-17T11:07:27.0785240Z SIGABRT(6), PID: 1828347, Thread 1828602: 
2025-07-17T11:07:27.0785707Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0785914Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0786115Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0786310Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0786804Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0787081Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0787299Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0787303Z 
2025-07-17T11:07:27.0787389Z SIGABRT(6), PID: 1828347, Thread 1828606: 
2025-07-17T11:07:27.0787920Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0788125Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0788335Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0788591Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0789088Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0789294Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0789501Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0789507Z 
2025-07-17T11:07:27.0789597Z SIGABRT(6), PID: 1828347, Thread 1828611: 
2025-07-17T11:07:27.0790059Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0790263Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0790467Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0790663Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0791154Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0791355Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0791566Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0791572Z 
2025-07-17T11:07:27.0791721Z SIGABRT(6), PID: 1828344, Thread 1833235: 
2025-07-17T11:07:27.0792193Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0792397Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0792601Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0792798Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0793174Z frame #4: <unknown function> + 0x5a02023 (0x7f9e8f7c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0793740Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9e8f7c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0794338Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9e8f7bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0794718Z frame #7: <unknown function> + 0x79d655 (0x7f9ea0ca7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0794947Z frame #8: <unknown function> + 0xdc253 (0x7f9e57cf6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0795224Z frame #9: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0795437Z frame #10: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0795441Z 
2025-07-17T11:07:27.0795531Z SIGABRT(6), PID: 1828347, Thread 1828616: 
2025-07-17T11:07:27.0795998Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0796268Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0796470Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0796674Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0797233Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0797445Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0797653Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0797657Z 
2025-07-17T11:07:27.0797746Z SIGABRT(6), PID: 1828347, Thread 1828619: 
2025-07-17T11:07:27.0798213Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0798419Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0798619Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0798817Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0799307Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0799516Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0799721Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0799729Z 
2025-07-17T11:07:27.0799818Z SIGABRT(6), PID: 1828346, Thread 1833208: 
2025-07-17T11:07:27.0800361Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0800570Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0800773Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0800969Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0801339Z frame #4: <unknown function> + 0x5a02023 (0x7f428f69b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0801898Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f428f69f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0802487Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f428f6961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0802866Z frame #7: <unknown function> + 0x79d655 (0x7f42a0b86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0803092Z frame #8: <unknown function> + 0xdc253 (0x7f4257bd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0803297Z frame #9: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0803572Z frame #10: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0803579Z 
2025-07-17T11:07:27.0803666Z SIGABRT(6), PID: 1828347, Thread 1828623: 
2025-07-17T11:07:27.0804130Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0804399Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0804602Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0804797Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0805354Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0805562Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0805772Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0805776Z 
2025-07-17T11:07:27.0805862Z SIGABRT(6), PID: 1828347, Thread 1828627: 
2025-07-17T11:07:27.0806336Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0806545Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0806748Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0806942Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0807441Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0807646Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0807856Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0807859Z 
2025-07-17T11:07:27.0807947Z SIGABRT(6), PID: 1828347, Thread 1828631: 
2025-07-17T11:07:27.0808486Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0808691Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0808895Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0809090Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0809585Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0809790Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0809998Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0810004Z 
2025-07-17T11:07:27.0810090Z SIGABRT(6), PID: 1828347, Thread 1828635: 
2025-07-17T11:07:27.0810557Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0810759Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0810963Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0811156Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0811713Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0811917Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0812126Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0812190Z 
2025-07-17T11:07:27.0812283Z SIGABRT(6), PID: 1828344, Thread 1833236: 
2025-07-17T11:07:27.0812751Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0812955Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0813218Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0813417Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0813789Z frame #4: <unknown function> + 0x5a02023 (0x7f9e8f7c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0814352Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9e8f7c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0814944Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9e8f7bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0815324Z frame #7: <unknown function> + 0x79d655 (0x7f9ea0ca7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0815548Z frame #8: <unknown function> + 0xdc253 (0x7f9e57cf6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0815753Z frame #9: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0816029Z frame #10: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0816033Z 
2025-07-17T11:07:27.0816125Z SIGABRT(6), PID: 1828347, Thread 1828639: 
2025-07-17T11:07:27.0816680Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0816891Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0817091Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0817293Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0817795Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0818002Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0818208Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0818212Z 
2025-07-17T11:07:27.0818303Z SIGABRT(6), PID: 1828347, Thread 1828642: 
2025-07-17T11:07:27.0818774Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0818980Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0819180Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0819378Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0819957Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0820162Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0820366Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0820442Z 
2025-07-17T11:07:27.0820536Z SIGABRT(6), PID: 1828346, Thread 1833209: 
2025-07-17T11:07:27.0820998Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0821204Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0821476Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0821675Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0822044Z frame #4: <unknown function> + 0x5a02023 (0x7f428f69b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0822608Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f428f69f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0823197Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f428f6961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0823574Z frame #7: <unknown function> + 0x79d655 (0x7f42a0b86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0823799Z frame #8: <unknown function> + 0xdc253 (0x7f4257bd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0824005Z frame #9: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0824215Z frame #10: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0824222Z 
2025-07-17T11:07:27.0824308Z SIGABRT(6), PID: 1828347, Thread 1828646: 
2025-07-17T11:07:27.0824837Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0825046Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0825253Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0825449Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0825947Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0826153Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0826361Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0826365Z 
2025-07-17T11:07:27.0826453Z SIGABRT(6), PID: 1828347, Thread 1828651: 
2025-07-17T11:07:27.0826927Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0827141Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0827344Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0827541Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0828033Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0828305Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0828516Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0828580Z 
2025-07-17T11:07:27.0828671Z SIGABRT(6), PID: 1828347, Thread 1828655: 
2025-07-17T11:07:27.0829139Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0829341Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0829624Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0829823Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0830318Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0830522Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0830728Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0830734Z 
2025-07-17T11:07:27.0830821Z SIGABRT(6), PID: 1828344, Thread 1833237: 
2025-07-17T11:07:27.0831289Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0831492Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0831700Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0831895Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0832264Z frame #4: <unknown function> + 0x5a02023 (0x7f9e8f7c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0832819Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9e8f7c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0833479Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9e8f7bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0833864Z frame #7: <unknown function> + 0x79d655 (0x7f9ea0ca7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0834091Z frame #8: <unknown function> + 0xdc253 (0x7f9e57cf6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0834300Z frame #9: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0834508Z frame #10: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0834512Z 
2025-07-17T11:07:27.0834601Z SIGABRT(6), PID: 1828347, Thread 1828658: 
2025-07-17T11:07:27.0835065Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0835274Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0835475Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0835673Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0836168Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0836443Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0836652Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0836656Z 
2025-07-17T11:07:27.0836745Z SIGABRT(6), PID: 1828347, Thread 1828663: 
2025-07-17T11:07:27.0837275Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0837481Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0837685Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0837942Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0838437Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0838645Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0838848Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0838853Z 
2025-07-17T11:07:27.0838943Z SIGABRT(6), PID: 1828347, Thread 1828667: 
2025-07-17T11:07:27.0839405Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0839609Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0839810Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0840004Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0840499Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0840705Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0840913Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0840921Z 
2025-07-17T11:07:27.0841072Z SIGABRT(6), PID: 1828346, Thread 1833210: 
2025-07-17T11:07:27.0841538Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0841742Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0841943Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0842141Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0842517Z frame #4: <unknown function> + 0x5a02023 (0x7f428f69b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0843079Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f428f69f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0843665Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f428f6961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0844043Z frame #7: <unknown function> + 0x79d655 (0x7f42a0b86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0844271Z frame #8: <unknown function> + 0xdc253 (0x7f4257bd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0844546Z frame #9: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0844758Z frame #10: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0844763Z 
2025-07-17T11:07:27.0844852Z SIGABRT(6), PID: 1828347, Thread 1828671: 
2025-07-17T11:07:27.0845325Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0845594Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0845798Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0846054Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0846559Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0846766Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0846982Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0846987Z 
2025-07-17T11:07:27.0847074Z SIGABRT(6), PID: 1828347, Thread 1828673: 
2025-07-17T11:07:27.0847541Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0847742Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0847946Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0848142Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0848637Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0848840Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0849050Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0849056Z 
2025-07-17T11:07:27.0849210Z SIGABRT(6), PID: 1828347, Thread 1828676: 
2025-07-17T11:07:27.0849680Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0849882Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0850085Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0850279Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0850781Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0850994Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0851210Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0851216Z 
2025-07-17T11:07:27.0851302Z SIGABRT(6), PID: 1828347, Thread 1828680: 
2025-07-17T11:07:27.0851779Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0851987Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0852190Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0852458Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0852957Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0853161Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0853554Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0853558Z 
2025-07-17T11:07:27.0853646Z SIGABRT(6), PID: 1828344, Thread 1833238: 
2025-07-17T11:07:27.0854117Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0854385Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0854595Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0854796Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0855165Z frame #4: <unknown function> + 0x5a02023 (0x7f9e8f7c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0855734Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9e8f7c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0856386Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9e8f7bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0856773Z frame #7: <unknown function> + 0x79d655 (0x7f9ea0ca7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0857003Z frame #8: <unknown function> + 0xdc253 (0x7f9e57cf6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0857210Z frame #9: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0857420Z frame #10: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0857425Z 
2025-07-17T11:07:27.0857518Z SIGABRT(6), PID: 1828347, Thread 1828684: 
2025-07-17T11:07:27.0858067Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0858281Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0858482Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0858684Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0859182Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0859390Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0859596Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0859601Z 
2025-07-17T11:07:27.0859690Z SIGABRT(6), PID: 1828347, Thread 1828688: 
2025-07-17T11:07:27.0860162Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0860370Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0860571Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0860851Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0861346Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0861553Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0861860Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0861866Z 
2025-07-17T11:07:27.0861956Z SIGABRT(6), PID: 1828347, Thread 1828691: 
2025-07-17T11:07:27.0862420Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0862699Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0862900Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0863099Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0863590Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0863797Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0864004Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0864011Z 
2025-07-17T11:07:27.0864097Z SIGABRT(6), PID: 1828346, Thread 1833211: 
2025-07-17T11:07:27.0864562Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0864766Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0864968Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0865162Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0865531Z frame #4: <unknown function> + 0x5a02023 (0x7f428f69b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0866159Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f428f69f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0866756Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f428f6961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0867135Z frame #7: <unknown function> + 0x79d655 (0x7f42a0b86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0867372Z frame #8: <unknown function> + 0xdc253 (0x7f4257bd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0867576Z frame #9: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0867786Z frame #10: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0867791Z 
2025-07-17T11:07:27.0867878Z SIGABRT(6), PID: 1828347, Thread 1828695: 
2025-07-17T11:07:27.0868348Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0868555Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0868761Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0868955Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0869525Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0869730Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0869939Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0870002Z 
2025-07-17T11:07:27.0870094Z SIGABRT(6), PID: 1828347, Thread 1828700: 
2025-07-17T11:07:27.0870565Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0870768Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0871029Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0871229Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0871727Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0871932Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0872143Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0872147Z 
2025-07-17T11:07:27.0872233Z SIGABRT(6), PID: 1828347, Thread 1828703: 
2025-07-17T11:07:27.0872701Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0872905Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0873113Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0873306Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0873800Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0874007Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0874288Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0874293Z 
2025-07-17T11:07:27.0874384Z SIGABRT(6), PID: 1828344, Thread 1833240: 
2025-07-17T11:07:27.0874848Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9e59a89b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0875053Z frame #1: <unknown function> + 0x42520 (0x7f9f4b343520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0875256Z frame #2: <unknown function> + 0x91117 (0x7f9f4b392117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0875453Z frame #3: pthread_cond_wait + 0x211 (0x7f9f4b394a41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0875823Z frame #4: <unknown function> + 0x5a02023 (0x7f9e8f7c0023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0876390Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9e8f7c41ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0876978Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9e8f7bb1d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0877367Z frame #7: <unknown function> + 0x79d655 (0x7f9ea0ca7655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0877666Z frame #8: <unknown function> + 0xdc253 (0x7f9e57cf6253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0877876Z frame #9: <unknown function> + 0x94ac3 (0x7f9f4b395ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0878085Z frame #10: <unknown function> + 0x126850 (0x7f9f4b427850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0878149Z 
2025-07-17T11:07:27.0878244Z SIGABRT(6), PID: 1828347, Thread 1828707: 
2025-07-17T11:07:27.0878713Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0878920Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0879180Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0879384Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0879880Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0880085Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0880293Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0880299Z 
2025-07-17T11:07:27.0880388Z SIGABRT(6), PID: 1828347, Thread 1828711: 
2025-07-17T11:07:27.0880851Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0881057Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0881257Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0881455Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0881945Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0882152Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0882422Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0882429Z 
2025-07-17T11:07:27.0882519Z SIGABRT(6), PID: 1828347, Thread 1828715: 
2025-07-17T11:07:27.0882984Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0883192Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0883396Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0883593Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0884084Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0884293Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0884498Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0884504Z 
2025-07-17T11:07:27.0884590Z SIGABRT(6), PID: 1828346, Thread 1833212: 
2025-07-17T11:07:27.0885057Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0885258Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0885531Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0885724Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0886092Z frame #4: <unknown function> + 0x5a02023 (0x7f428f69b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0886716Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f428f69f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0887368Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f428f6961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0887752Z frame #7: <unknown function> + 0x79d655 (0x7f42a0b86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0887990Z frame #8: <unknown function> + 0xdc253 (0x7f4257bd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0888194Z frame #9: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0888403Z frame #10: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0888409Z 
2025-07-17T11:07:27.0888498Z SIGABRT(6), PID: 1828347, Thread 1828719: 
2025-07-17T11:07:27.0888966Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0889168Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0889372Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0889569Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0890061Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0890266Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0890543Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0890548Z 
2025-07-17T11:07:27.0890639Z SIGABRT(6), PID: 1828347, Thread 1828723: 
2025-07-17T11:07:27.0891109Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0891314Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0891517Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0891715Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0892211Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0892416Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0892627Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0892631Z 
2025-07-17T11:07:27.0892716Z SIGABRT(6), PID: 1828348, Thread 1828348: 
2025-07-17T11:07:27.0893183Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0893388Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0893721Z frame #2: <unknown function> + 0x7741f (0x7f9f0d1ce41f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0893949Z frame #3: <unknown function> + 0x76c8e (0x7f9f0d1cdc8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0894174Z frame #4: <unknown function> + 0x6ad69 (0x7f9f0d1c1d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0894388Z frame #5: <unknown function> + 0x3f54db (0x7f9f9de264db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0894664Z frame #6: <unknown function> + 0x3e11d8 (0x7f9f9de121d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0894871Z frame #7: <unknown function> + 0xb2f86 (0x7f9f9dae3f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0895079Z frame #8: <unknown function> + 0x8e501 (0x7f9f9dabf501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0895516Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f9fc88c67ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:27.0895965Z frame #10: <unknown function> + 0xd307d9 (0x7f9fe6e197d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0896193Z frame #11: <unknown function> + 0x139902 (0x55f38b6a4902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0896442Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55f38b69ba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0896673Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0896912Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55f38b69ba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0897138Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0897376Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0897599Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0897841Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0898059Z frame #19: <unknown function> + 0x1474b2 (0x55f38b6b24b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0898260Z frame #20: PyObject_Call + 0xbc (0x55f38b6b2e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0898564Z frame #21: <unknown function> + 0x1ff69d (0x55f38b76a69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0898796Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55f38b6a0023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0899034Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55f38b69bcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0899255Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0899487Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55f38b6973bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0899710Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0899942Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55f38b6973bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0900164Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0900402Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55f38b69ba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0900655Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55f38b69f4b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0900869Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55f38b6b0ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0901084Z frame #32: <unknown function> + 0x205949 (0x55f38b770949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0901304Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55f38b6a0023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0901625Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55f38b69bcc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0901845Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0902080Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55f38b6973bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0902377Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0902612Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55f38b6973bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0902830Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0903062Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55f38b6973bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0903350Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0903597Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55f38b6983f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0903817Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0904202Z frame #44: <unknown function> + 0x7ab3f7 (0x7f9fe68943f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0904421Z frame #45: <unknown function> + 0x13bc88 (0x55f38b6a6c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0904625Z frame #46: PyObject_Call + 0x20f (0x55f38b6b2f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0904867Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55f38b69c98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0905084Z frame #48: <unknown function> + 0x147608 (0x55f38b6b2608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0905320Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f38b699c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0905546Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0905778Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55f38b6973bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0906002Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0906301Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55f38b6983f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0906530Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0906763Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55f38b6973bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0906983Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0907220Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f38b699c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0907451Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0907685Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f38b699c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0907902Z frame #60: <unknown function> + 0x147608 (0x55f38b6b2608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0908141Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f38b699c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0908361Z frame #62: <unknown function> + 0x147608 (0x55f38b6b2608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0908365Z 
2025-07-17T11:07:27.0908457Z SIGABRT(6), PID: 1828347, Thread 1828727: 
2025-07-17T11:07:27.0908943Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0909160Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0909436Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0909642Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0910150Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0910428Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0910639Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0910644Z 
2025-07-17T11:07:27.0910736Z SIGABRT(6), PID: 1828348, Thread 1828592: 
2025-07-17T11:07:27.0911270Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0911483Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0911685Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0911886Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0912386Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0912595Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0912799Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0912803Z 
2025-07-17T11:07:27.0912892Z SIGABRT(6), PID: 1828347, Thread 1828731: 
2025-07-17T11:07:27.0913358Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0913569Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0913771Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0913970Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0914525Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0914738Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0914945Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0914950Z 
2025-07-17T11:07:27.0915041Z SIGABRT(6), PID: 1828347, Thread 1828735: 
2025-07-17T11:07:27.0915505Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0915715Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0915919Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0916118Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0916613Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0916822Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0917031Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0917035Z 
2025-07-17T11:07:27.0917201Z SIGABRT(6), PID: 1828348, Thread 1828596: 
2025-07-17T11:07:27.0917663Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0917871Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0918071Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0918335Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0918827Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0919032Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0919296Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0919306Z 
2025-07-17T11:07:27.0919395Z SIGABRT(6), PID: 1828349, Thread 1828349: 
2025-07-17T11:07:27.0919859Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0920061Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0920299Z frame #2: <unknown function> + 0x7741f (0x7f8bb2f2f41f in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0920528Z frame #3: <unknown function> + 0x76c8e (0x7f8bb2f2ec8e in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0920750Z frame #4: <unknown function> + 0x6ad69 (0x7f8bb2f22d69 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.0920962Z frame #5: <unknown function> + 0x3f54db (0x7f8c43b874db in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0921174Z frame #6: <unknown function> + 0x3e11d8 (0x7f8c43b731d8 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0921381Z frame #7: <unknown function> + 0xb2f86 (0x7f8c43844f86 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0921587Z frame #8: <unknown function> + 0x8e501 (0x7f8c43820501 in /opt/rocm/lib/libamdhip64.so.6)
2025-07-17T11:07:27.0921965Z frame #9: c10::hip::device_synchronize() + 0x5a (0x7f8c6e6277ca in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so)
2025-07-17T11:07:27.0922436Z frame #10: <unknown function> + 0xd307d9 (0x7f8c8cb767d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0922662Z frame #11: <unknown function> + 0x139902 (0x55d832bb5902 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0922913Z frame #12: _PyEval_EvalFrameDefault + 0x49b5 (0x55d832baca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0923139Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0923379Z frame #14: _PyEval_EvalFrameDefault + 0x49b5 (0x55d832baca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0923600Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0923837Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0924055Z frame #17: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0924292Z frame #18: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0924509Z frame #19: <unknown function> + 0x1474b2 (0x55d832bc34b2 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0924711Z frame #20: PyObject_Call + 0xbc (0x55d832bc3e4c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0924932Z frame #21: <unknown function> + 0x1ff69d (0x55d832c7b69d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0925157Z frame #22: _PyObject_MakeTpCall + 0x2d3 (0x55d832bb1023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0925469Z frame #23: _PyEval_EvalFrameDefault + 0x4c16 (0x55d832baccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0925693Z frame #24: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0925926Z frame #25: _PyEval_EvalFrameDefault + 0x30c (0x55d832ba83bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0926209Z frame #26: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0926439Z frame #27: _PyEval_EvalFrameDefault + 0x30c (0x55d832ba83bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0926659Z frame #28: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0926958Z frame #29: _PyEval_EvalFrameDefault + 0x49b5 (0x55d832baca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0927218Z frame #30: _PyObject_FastCallDictTstate + 0xd0 (0x55d832bb04b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0927439Z frame #31: _PyObject_Call_Prepend + 0x69 (0x55d832bc1ab9 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0927653Z frame #32: <unknown function> + 0x205949 (0x55d832c81949 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0927876Z frame #33: _PyObject_MakeTpCall + 0x2d3 (0x55d832bb1023 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0928122Z frame #34: _PyEval_EvalFrameDefault + 0x4c16 (0x55d832baccc6 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0928341Z frame #35: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0928575Z frame #36: _PyEval_EvalFrameDefault + 0x30c (0x55d832ba83bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0928795Z frame #37: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0929026Z frame #38: _PyEval_EvalFrameDefault + 0x30c (0x55d832ba83bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0929247Z frame #39: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0929480Z frame #40: _PyEval_EvalFrameDefault + 0x30c (0x55d832ba83bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0929699Z frame #41: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0930007Z frame #42: _PyEval_EvalFrameDefault + 0x1340 (0x55d832ba93f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0930229Z frame #43: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0930614Z frame #44: <unknown function> + 0x7ab3f7 (0x7f8c8c5f13f7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0930834Z frame #45: <unknown function> + 0x13bc88 (0x55d832bb7c88 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0931037Z frame #46: PyObject_Call + 0x20f (0x55d832bc3f9f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0931279Z frame #47: _PyEval_EvalFrameDefault + 0x58dd (0x55d832bad98d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0931494Z frame #48: <unknown function> + 0x147608 (0x55d832bc3608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0931729Z frame #49: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d832baac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0931959Z frame #50: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0932190Z frame #51: _PyEval_EvalFrameDefault + 0x30c (0x55d832ba83bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0932409Z frame #52: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0932643Z frame #53: _PyEval_EvalFrameDefault + 0x1340 (0x55d832ba93f0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0932864Z frame #54: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0933162Z frame #55: _PyEval_EvalFrameDefault + 0x30c (0x55d832ba83bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0933387Z frame #56: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0933623Z frame #57: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d832baac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0933908Z frame #58: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0934143Z frame #59: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d832baac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0934359Z frame #60: <unknown function> + 0x147608 (0x55d832bc3608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0934657Z frame #61: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d832baac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0934876Z frame #62: <unknown function> + 0x147608 (0x55d832bc3608 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.0934969Z SIGABRT(6), PID: 1828347, Thread 1828738: 
2025-07-17T11:07:27.0935445Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0935655Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0935869Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0936137Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0936640Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0936847Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0937061Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0937065Z 
2025-07-17T11:07:27.0937070Z 
2025-07-17T11:07:27.0937157Z SIGABRT(6), PID: 1828348, Thread 1828600: 
2025-07-17T11:07:27.0937626Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0937913Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0938125Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0938321Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0938819Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0939021Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0939232Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0939236Z 
2025-07-17T11:07:27.0939322Z SIGABRT(6), PID: 1828347, Thread 1828741: 
2025-07-17T11:07:27.0939794Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0940001Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0940203Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0940399Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0940896Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0941185Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0941398Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0941402Z 
2025-07-17T11:07:27.0941487Z SIGABRT(6), PID: 1828349, Thread 1828794: 
2025-07-17T11:07:27.0941958Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0942235Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0942439Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0942639Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0943204Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0943414Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0943624Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0943629Z 
2025-07-17T11:07:27.0943715Z SIGABRT(6), PID: 1828348, Thread 1828604: 
2025-07-17T11:07:27.0944190Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0944395Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0944599Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0944797Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0945289Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0945494Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0945697Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0945702Z 
2025-07-17T11:07:27.0945853Z SIGABRT(6), PID: 1828347, Thread 1828745: 
2025-07-17T11:07:27.0946321Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0946529Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0946731Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0946925Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0947426Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0947634Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0947839Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0947847Z 
2025-07-17T11:07:27.0947935Z SIGABRT(6), PID: 1828346, Thread 1833213: 
2025-07-17T11:07:27.0948400Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0948606Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0948806Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0949069Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0949440Z frame #4: <unknown function> + 0x5a02023 (0x7f428f69b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0950010Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f428f69f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0950655Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f428f6961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0951117Z frame #7: <unknown function> + 0x79d655 (0x7f42a0b86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.0951350Z frame #8: <unknown function> + 0xdc253 (0x7f4257bd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.0951558Z frame #9: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0951768Z frame #10: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0951772Z 
2025-07-17T11:07:27.0951863Z SIGABRT(6), PID: 1828349, Thread 1828795: 
2025-07-17T11:07:27.0952335Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0952544Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0952748Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0952950Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0953448Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0953660Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0953864Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0953868Z 
2025-07-17T11:07:27.0953960Z SIGABRT(6), PID: 1828347, Thread 1828750: 
2025-07-17T11:07:27.0954493Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0954702Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0954911Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0955106Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0955602Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0955809Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0956018Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0956107Z SIGABRT(6), PID: 1828348, Thread 1828608: 
2025-07-17T11:07:27.0956573Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0956776Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0956978Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0957181Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0957752Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0957957Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0958162Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0958230Z 
2025-07-17T11:07:27.0958233Z 
2025-07-17T11:07:27.0958321Z SIGABRT(6), PID: 1828349, Thread 1828796: 
2025-07-17T11:07:27.0958791Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0959057Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0959267Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0959462Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0959953Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0960156Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0960367Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0960371Z 
2025-07-17T11:07:27.0960460Z SIGABRT(6), PID: 1828347, Thread 1828753: 
2025-07-17T11:07:27.0960925Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0961131Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0961335Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0961528Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0962020Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0962287Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0962501Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0962506Z 
2025-07-17T11:07:27.0962591Z SIGABRT(6), PID: 1828348, Thread 1828612: 
2025-07-17T11:07:27.0963060Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0963265Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0963471Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0963661Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0964151Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0964360Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0964571Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0964575Z 
2025-07-17T11:07:27.0964660Z SIGABRT(6), PID: 1828347, Thread 1828757: 
2025-07-17T11:07:27.0965126Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0965400Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0965602Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0965798Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0966292Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0966565Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0966773Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0966777Z 
2025-07-17T11:07:27.0966937Z SIGABRT(6), PID: 1828349, Thread 1828797: 
2025-07-17T11:07:27.0967404Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0967613Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0967814Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0968008Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0968503Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0968712Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0968916Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0968921Z 
2025-07-17T11:07:27.0969012Z SIGABRT(6), PID: 1828348, Thread 1828615: 
2025-07-17T11:07:27.0969477Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0969683Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0969888Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0970151Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0970651Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0970859Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0971064Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0971070Z 
2025-07-17T11:07:27.0971161Z SIGABRT(6), PID: 1828347, Thread 1828760: 
2025-07-17T11:07:27.0971622Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0971828Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0972027Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0972226Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0972715Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0972923Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0973126Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0973197Z 
2025-07-17T11:07:27.0973289Z SIGABRT(6), PID: 1828349, Thread 1828798: 
2025-07-17T11:07:27.0973748Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0973953Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0974223Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0974418Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0974905Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0975170Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0975382Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0975389Z 
2025-07-17T11:07:27.0975475Z SIGABRT(6), PID: 1828348, Thread 1828618: 
2025-07-17T11:07:27.0976001Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0976209Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0976412Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0976604Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0977105Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0977309Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0977519Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0977523Z 
2025-07-17T11:07:27.0977611Z SIGABRT(6), PID: 1828347, Thread 1828764: 
2025-07-17T11:07:27.0978075Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0978360Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0978567Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0978761Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0979257Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0979461Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0979669Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0979673Z 
2025-07-17T11:07:27.0979762Z SIGABRT(6), PID: 1828349, Thread 1828799: 
2025-07-17T11:07:27.0980232Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0980441Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0980643Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0980834Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0981327Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0981718Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0981927Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0981933Z 
2025-07-17T11:07:27.0982020Z SIGABRT(6), PID: 1828348, Thread 1828622: 
2025-07-17T11:07:27.0982612Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0982827Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0983034Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0983304Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0983819Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0984029Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0984240Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0984246Z 
2025-07-17T11:07:27.0984336Z SIGABRT(6), PID: 1828347, Thread 1828766: 
2025-07-17T11:07:27.0984810Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0985017Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0985222Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0985418Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0985917Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0986121Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0986332Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0986403Z 
2025-07-17T11:07:27.0986494Z SIGABRT(6), PID: 1828349, Thread 1828800: 
2025-07-17T11:07:27.0986962Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0987168Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0987370Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0987577Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0988071Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0988285Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0988494Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0988498Z 
2025-07-17T11:07:27.0988591Z SIGABRT(6), PID: 1828348, Thread 1828626: 
2025-07-17T11:07:27.0989062Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0989271Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0989542Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0989741Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0990238Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0990511Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0990716Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0990720Z 
2025-07-17T11:07:27.0990810Z SIGABRT(6), PID: 1828347, Thread 1828769: 
2025-07-17T11:07:27.0991336Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0991546Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0991750Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0991947Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0992440Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0992650Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0992854Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0992858Z 
2025-07-17T11:07:27.0992948Z SIGABRT(6), PID: 1828349, Thread 1828801: 
2025-07-17T11:07:27.0993412Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0993621Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0993821Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0994016Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0994567Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0994783Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0994988Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0994992Z 
2025-07-17T11:07:27.0995083Z SIGABRT(6), PID: 1828348, Thread 1828630: 
2025-07-17T11:07:27.0995550Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0995758Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0995960Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0996156Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0996648Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.0996853Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0997057Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0997063Z 
2025-07-17T11:07:27.0997151Z SIGABRT(6), PID: 1828346, Thread 1833214: 
2025-07-17T11:07:27.0997691Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.0997895Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0998097Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0998356Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.0998735Z frame #4: <unknown function> + 0x5a02023 (0x7f428f69b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0999297Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f428f69f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.0999949Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f428f6961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1000341Z frame #7: <unknown function> + 0x79d655 (0x7f42a0b86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1000573Z frame #8: <unknown function> + 0xdc253 (0x7f4257bd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1000781Z frame #9: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1000995Z frame #10: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1001000Z 
2025-07-17T11:07:27.1001088Z SIGABRT(6), PID: 1828347, Thread 1828772: 
2025-07-17T11:07:27.1001559Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1001765Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1001968Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1002163Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1002661Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1002929Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1003144Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1003148Z 
2025-07-17T11:07:27.1003237Z SIGABRT(6), PID: 1828349, Thread 1828802: 
2025-07-17T11:07:27.1003707Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1003915Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1004118Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1004322Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1004829Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1005039Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1005251Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1005255Z 
2025-07-17T11:07:27.1005342Z SIGABRT(6), PID: 1828347, Thread 1828775: 
2025-07-17T11:07:27.1005815Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1006087Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1006293Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1006488Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1007050Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1007259Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1007466Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1007530Z 
2025-07-17T11:07:27.1007623Z SIGABRT(6), PID: 1828348, Thread 1828633: 
2025-07-17T11:07:27.1008092Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1008301Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1008503Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1008703Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1009196Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1009403Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1009610Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1009615Z 
2025-07-17T11:07:27.1009704Z SIGABRT(6), PID: 1828349, Thread 1828803: 
2025-07-17T11:07:27.1010165Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1010373Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1010574Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1010839Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1011332Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1011541Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1011745Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1011750Z 
2025-07-17T11:07:27.1011840Z SIGABRT(6), PID: 1828347, Thread 1828778: 
2025-07-17T11:07:27.1012301Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1012505Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1012710Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1012907Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1013400Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1013611Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1013908Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1013912Z 
2025-07-17T11:07:27.1014004Z SIGABRT(6), PID: 1828348, Thread 1828637: 
2025-07-17T11:07:27.1014468Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1014743Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1014948Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1015146Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1015698Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1015972Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1016179Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1016183Z 
2025-07-17T11:07:27.1016273Z SIGABRT(6), PID: 1828349, Thread 1828804: 
2025-07-17T11:07:27.1016735Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1016944Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1017144Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1017341Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1017834Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1018038Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1018245Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1018248Z 
2025-07-17T11:07:27.1018334Z SIGABRT(6), PID: 1828347, Thread 1828781: 
2025-07-17T11:07:27.1018878Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1019087Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1019291Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1019486Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1019979Z frame #4: <unknown function> + 0x3475ab (0x7f1adc2085ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1020185Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1020390Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1020476Z SIGABRT(6), PID: 1828348, Thread 1828643: 
2025-07-17T11:07:27.1020944Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1021146Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1021351Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1021545Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1022039Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1022325Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1022533Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1022613Z 
2025-07-17T11:07:27.1022616Z 
2025-07-17T11:07:27.1022705Z SIGABRT(6), PID: 1828347, Thread 1829402: 
2025-07-17T11:07:27.1023173Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1023622Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f19eda6508a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1023900Z frame #2: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1024096Z frame #3: pthread_kill + 0x12c (0x7f1adf3729fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1024262Z frame #4: raise + 0x16 (0x7f1adf31e476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1024418Z frame #5: abort + 0xd3 (0x7f1adf3047f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1024654Z frame #6: <unknown function> + 0x972cc (0x7f195b5ee2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1024884Z frame #7: <unknown function> + 0x95f47 (0x7f195b5ecf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1025109Z frame #8: <unknown function> + 0x34b71 (0x7f195b58bb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1025315Z frame #9: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1025529Z frame #10: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1025533Z 
2025-07-17T11:07:27.1025620Z SIGABRT(6), PID: 1828349, Thread 1828805: 
2025-07-17T11:07:27.1026099Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1026301Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1026505Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1026767Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1027267Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1027471Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1027687Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1027693Z 
2025-07-17T11:07:27.1027781Z SIGABRT(6), PID: 1828348, Thread 1828647: 
2025-07-17T11:07:27.1028252Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1028459Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1028665Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1028860Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1029356Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1029563Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1029768Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1029837Z 
2025-07-17T11:07:27.1029927Z SIGABRT(6), PID: 1828347, Thread 1832530: 
2025-07-17T11:07:27.1030391Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1030596Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1030830Z frame #2: __poll + 0x4f (0x7f1adf3f4bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1031243Z frame #3: <unknown function> + 0x489f (0x7f1adef0089f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:27.1031473Z frame #4: <unknown function> + 0x1483bc (0x55f4795033bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1031779Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1032005Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1032239Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1032459Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1032704Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55f4794eba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1032933Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1033179Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f4794e9c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1033401Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1033638Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1033860Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1034096Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1034313Z frame #16: <unknown function> + 0x147734 (0x55f479502734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1034532Z frame #17: <unknown function> + 0x21ca82 (0x55f4795d7a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1034811Z frame #18: <unknown function> + 0x21c9e4 (0x55f4795d79e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1035026Z frame #19: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1035237Z frame #20: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1035330Z SIGABRT(6), PID: 1828349, Thread 1828806: 
2025-07-17T11:07:27.1035808Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1036021Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1036224Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1036426Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1036924Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1037130Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1037343Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1037349Z 
2025-07-17T11:07:27.1037353Z 
2025-07-17T11:07:27.1037444Z SIGABRT(6), PID: 1828348, Thread 1828650: 
2025-07-17T11:07:27.1037981Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1038187Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1038390Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1038652Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1039147Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1039353Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1039615Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1039621Z 
2025-07-17T11:07:27.1039712Z SIGABRT(6), PID: 1828349, Thread 1828807: 
2025-07-17T11:07:27.1040179Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1040387Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1040593Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1040791Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1041283Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1041490Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1041695Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1041701Z 
2025-07-17T11:07:27.1041790Z SIGABRT(6), PID: 1828348, Thread 1828654: 
2025-07-17T11:07:27.1042255Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1042460Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1042745Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1042945Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1043436Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1043644Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1043847Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1043858Z 
2025-07-17T11:07:27.1043944Z SIGABRT(6), PID: 1828346, Thread 1833215: 
2025-07-17T11:07:27.1044414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f4259964b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1044619Z frame #1: <unknown function> + 0x42520 (0x7f434b21e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1044833Z frame #2: <unknown function> + 0x91117 (0x7f434b26d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1045025Z frame #3: pthread_cond_wait + 0x211 (0x7f434b26fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1045399Z frame #4: <unknown function> + 0x5a02023 (0x7f428f69b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1045965Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f428f69f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1046624Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f428f6961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1047074Z frame #7: <unknown function> + 0x79d655 (0x7f42a0b86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1047306Z frame #8: <unknown function> + 0xdc253 (0x7f4257bd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1047509Z frame #9: <unknown function> + 0x94ac3 (0x7f434b270ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1047784Z frame #10: <unknown function> + 0x126850 (0x7f434b302850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1047789Z 
2025-07-17T11:07:27.1047881Z SIGABRT(6), PID: 1828349, Thread 1828808: 
2025-07-17T11:07:27.1048354Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1048558Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1048765Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1048964Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1049458Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1049662Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1049870Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1049876Z 
2025-07-17T11:07:27.1049962Z SIGABRT(6), PID: 1828348, Thread 1828659: 
2025-07-17T11:07:27.1050431Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1050636Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1050904Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1051098Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1051590Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1051794Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1052003Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1052007Z 
2025-07-17T11:07:27.1052094Z SIGABRT(6), PID: 1828347, Thread 1832531: 
2025-07-17T11:07:27.1052568Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1052770Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1052977Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1053195Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f1adf3702dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1053640Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f1a197ad048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.1054058Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f1a197afbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.1054361Z frame #6: <unknown function> + 0xdc253 (0x7f19ebcd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1054570Z frame #7: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1054777Z frame #8: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1054842Z 
2025-07-17T11:07:27.1054935Z SIGABRT(6), PID: 1828348, Thread 1828662: 
2025-07-17T11:07:27.1055401Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1055607Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1055867Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1056140Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1056639Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1056846Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1057054Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1057058Z 
2025-07-17T11:07:27.1057147Z SIGABRT(6), PID: 1828349, Thread 1828809: 
2025-07-17T11:07:27.1057615Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1057821Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1058021Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1058221Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1058710Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1058918Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1059207Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1059212Z 
2025-07-17T11:07:27.1059303Z SIGABRT(6), PID: 1828348, Thread 1828666: 
2025-07-17T11:07:27.1059771Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1059978Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1060188Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1060384Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1060874Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1061086Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1061290Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1061295Z 
2025-07-17T11:07:27.1061383Z SIGABRT(6), PID: 1828349, Thread 1828810: 
2025-07-17T11:07:27.1061852Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1062132Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1062339Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1062535Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1063024Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1063309Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1063515Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1063520Z 
2025-07-17T11:07:27.1063609Z SIGABRT(6), PID: 1828347, Thread 1832532: 
2025-07-17T11:07:27.1064148Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1064359Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1064559Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1064781Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f1adf3702dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1065254Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f1a197a7741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.1065481Z frame #5: <unknown function> + 0xdc253 (0x7f19ebcd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1065689Z frame #6: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1065895Z frame #7: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1065899Z 
2025-07-17T11:07:27.1065989Z SIGABRT(6), PID: 1828348, Thread 1828670: 
2025-07-17T11:07:27.1066450Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1066657Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1066860Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1067123Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1067628Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1067837Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1068041Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1068046Z 
2025-07-17T11:07:27.1068136Z SIGABRT(6), PID: 1828347, Thread 1832535: 
2025-07-17T11:07:27.1068599Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1068806Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1068976Z frame #2: ioctl + 0x3f (0x7f1adf3f694f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1069209Z frame #3: <unknown function> + 0x13eef0 (0x7f195b695ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1069434Z frame #4: <unknown function> + 0x1378c4 (0x7f195b68e8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1069663Z frame #5: <unknown function> + 0x95b2b (0x7f195b5ecb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1069886Z frame #6: <unknown function> + 0x34b71 (0x7f195b58bb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1070165Z frame #7: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1070373Z frame #8: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1070464Z SIGABRT(6), PID: 1828349, Thread 1828811: 
2025-07-17T11:07:27.1070930Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1071200Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1071401Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1071606Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1072181Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1072392Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1072598Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1072692Z SIGABRT(6), PID: 1828348, Thread 1828675: 
2025-07-17T11:07:27.1073156Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1073362Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1073562Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1073760Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1074252Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1074461Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1074661Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1074665Z 
2025-07-17T11:07:27.1074669Z 
2025-07-17T11:07:27.1074673Z 
2025-07-17T11:07:27.1074764Z SIGABRT(6), PID: 1828349, Thread 1828812: 
2025-07-17T11:07:27.1075290Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1075497Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1075699Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1075897Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1076389Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1076597Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1076801Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1076810Z 
2025-07-17T11:07:27.1076897Z SIGABRT(6), PID: 1828348, Thread 1828678: 
2025-07-17T11:07:27.1077374Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1077576Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1077782Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1078041Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1078536Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1078740Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1079015Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1079020Z 
2025-07-17T11:07:27.1079104Z SIGABRT(6), PID: 1828347, Thread 1833152: 
2025-07-17T11:07:27.1079574Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1079836Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1080009Z frame #2: __poll + 0x4f (0x7f1adf3f4bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1080219Z frame #3: <unknown function> + 0xd4540 (0x7f19a3805540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.1080425Z frame #4: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1080629Z frame #5: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1080635Z 
2025-07-17T11:07:27.1080729Z SIGABRT(6), PID: 1828349, Thread 1828813: 
2025-07-17T11:07:27.1081192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1081399Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1081601Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1081798Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1082291Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1082501Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1082705Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1082773Z 
2025-07-17T11:07:27.1082869Z SIGABRT(6), PID: 1828348, Thread 1828681: 
2025-07-17T11:07:27.1083335Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1083542Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1083741Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1083938Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1084426Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1084635Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1084840Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1084843Z 
2025-07-17T11:07:27.1084931Z SIGABRT(6), PID: 1828347, Thread 1833160: 
2025-07-17T11:07:27.1085396Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1085604Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1085843Z frame #2: __poll + 0x4f (0x7f1adf3f4bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1086058Z frame #3: <unknown function> + 0xd65a2 (0x7f19a38075a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.1086259Z frame #4: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1086466Z frame #5: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1086536Z 
2025-07-17T11:07:27.1086626Z SIGABRT(6), PID: 1828349, Thread 1828814: 
2025-07-17T11:07:27.1087096Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1087298Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1087561Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1087758Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1088253Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1088461Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1088665Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1088671Z 
2025-07-17T11:07:27.1088761Z SIGABRT(6), PID: 1828347, Thread 1833172: 
2025-07-17T11:07:27.1089228Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1089437Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1089635Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1089831Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1090042Z frame #4: <unknown function> + 0xd1e7f (0x7f19a3802e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.1090247Z frame #5: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1090459Z frame #6: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1090526Z 
2025-07-17T11:07:27.1090620Z SIGABRT(6), PID: 1828348, Thread 1828685: 
2025-07-17T11:07:27.1091086Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1091292Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1091493Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1091691Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1092184Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1092391Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1092596Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1092600Z 
2025-07-17T11:07:27.1092689Z SIGABRT(6), PID: 1828349, Thread 1828815: 
2025-07-17T11:07:27.1093155Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1093364Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1093635Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1093832Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1094322Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1094593Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1094798Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1094802Z 
2025-07-17T11:07:27.1094893Z SIGABRT(6), PID: 1828348, Thread 1828689: 
2025-07-17T11:07:27.1095419Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1095629Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1095830Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1096107Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1096599Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1096814Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1097019Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1097024Z 
2025-07-17T11:07:27.1097114Z SIGABRT(6), PID: 1828347, Thread 1833179: 
2025-07-17T11:07:27.1097581Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1097788Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1097987Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1098192Z frame #3: <unknown function> + 0x9ca53 (0x7f1adf378a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1098445Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55f4794cfbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1103498Z frame #5: <unknown function> + 0x1d314f (0x55f47958e14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1103787Z frame #6: <unknown function> + 0x1d2f6b (0x55f47958df6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1104026Z frame #7: <unknown function> + 0x6df36 (0x55f479428f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1104281Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1104519Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1104757Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1104985Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1105219Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1105449Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1105678Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1105900Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1106132Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1106443Z frame #17: <unknown function> + 0x147734 (0x55f479502734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1106660Z frame #18: <unknown function> + 0x21ca82 (0x55f4795d7a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1106879Z frame #19: <unknown function> + 0x21c9e4 (0x55f4795d79e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1107095Z frame #20: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1107438Z frame #21: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1107443Z 
2025-07-17T11:07:27.1107542Z SIGABRT(6), PID: 1828349, Thread 1828816: 
2025-07-17T11:07:27.1108060Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1108354Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1108574Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1108789Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1109321Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1109536Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1109750Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1109754Z 
2025-07-17T11:07:27.1109850Z SIGABRT(6), PID: 1828348, Thread 1828693: 
2025-07-17T11:07:27.1110337Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1110550Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1110752Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1110956Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1111456Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1111851Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1112067Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1112072Z 
2025-07-17T11:07:27.1112162Z SIGABRT(6), PID: 1828349, Thread 1828817: 
2025-07-17T11:07:27.1112633Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1112842Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1113044Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1113241Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1113733Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1113942Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1114147Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1114151Z 
2025-07-17T11:07:27.1114241Z SIGABRT(6), PID: 1828347, Thread 1833191: 
2025-07-17T11:07:27.1114709Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1114987Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1115153Z frame #2: read + 0x4c (0x7f1adf3f081c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1115334Z frame #3: _Py_read + 0x4c (0x55f479595a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1115626Z frame #4: <unknown function> + 0x208c3b (0x55f4795c3c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1115846Z frame #5: <unknown function> + 0x14648d (0x55f47950148d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1116088Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55f479510215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1116308Z frame #7: <unknown function> + 0x208ae7 (0x55f4795c3ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1116580Z frame #8: <unknown function> + 0x208472 (0x55f4795c3472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1116806Z frame #9: <unknown function> + 0x1da6ee (0x55f4795956ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1117029Z frame #10: <unknown function> + 0x1483bc (0x55f4795033bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1117272Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1117506Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1117744Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55f4794e73bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1117959Z frame #14: <unknown function> + 0x147734 (0x55f479502734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1118207Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f4794e9c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1118429Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1118664Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1118883Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55f4794f70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1119113Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55f4794e77b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1119327Z frame #20: <unknown function> + 0x147734 (0x55f479502734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1119611Z frame #21: <unknown function> + 0x21ca82 (0x55f4795d7a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1119830Z frame #22: <unknown function> + 0x21c9e4 (0x55f4795d79e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1120043Z frame #23: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1120256Z frame #24: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1120261Z 
2025-07-17T11:07:27.1120353Z SIGABRT(6), PID: 1828348, Thread 1828698: 
2025-07-17T11:07:27.1120824Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1121037Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1121242Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1121445Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1121940Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1122151Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1122358Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1122426Z 
2025-07-17T11:07:27.1122523Z SIGABRT(6), PID: 1828349, Thread 1828818: 
2025-07-17T11:07:27.1122995Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1123205Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1123472Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1123672Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1124165Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1124430Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1124642Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1124647Z 
2025-07-17T11:07:27.1124738Z SIGABRT(6), PID: 1828348, Thread 1828701: 
2025-07-17T11:07:27.1125207Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1125417Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1125618Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1125816Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1126314Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1126521Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1126729Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1126734Z 
2025-07-17T11:07:27.1126820Z SIGABRT(6), PID: 1828349, Thread 1828819: 
2025-07-17T11:07:27.1127288Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1127554Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1127765Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1127961Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1128457Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1128663Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1128871Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1128875Z 
2025-07-17T11:07:27.1128960Z SIGABRT(6), PID: 1828348, Thread 1828705: 
2025-07-17T11:07:27.1129428Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1129630Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1129835Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1130030Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1130521Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1130797Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1131006Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1131010Z 
2025-07-17T11:07:27.1131167Z SIGABRT(6), PID: 1828349, Thread 1828820: 
2025-07-17T11:07:27.1131640Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1131845Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1132048Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1132301Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1134633Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1134872Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1135097Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1135102Z 
2025-07-17T11:07:27.1135205Z SIGABRT(6), PID: 1828349, Thread 1828821: 
2025-07-17T11:07:27.1135707Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1136020Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1136241Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1136456Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1136989Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1137203Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1137414Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1137555Z 
2025-07-17T11:07:27.1137660Z SIGABRT(6), PID: 1828348, Thread 1828709: 
2025-07-17T11:07:27.1138145Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1138357Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1138562Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1138772Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1139273Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1139485Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1139694Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1139701Z 
2025-07-17T11:07:27.1139791Z SIGABRT(6), PID: 1828349, Thread 1828822: 
2025-07-17T11:07:27.1140262Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1140468Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1140766Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1140969Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1141461Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1141670Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1141882Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1141886Z 
2025-07-17T11:07:27.1141976Z SIGABRT(6), PID: 1828348, Thread 1828713: 
2025-07-17T11:07:27.1142517Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1142729Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1143061Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1143258Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1143756Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1143964Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1144176Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1144180Z 
2025-07-17T11:07:27.1144270Z SIGABRT(6), PID: 1828349, Thread 1828823: 
2025-07-17T11:07:27.1144745Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1144954Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1145160Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1145357Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1145917Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1146127Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1146339Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1146344Z 
2025-07-17T11:07:27.1146434Z SIGABRT(6), PID: 1828348, Thread 1828717: 
2025-07-17T11:07:27.1146913Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1147119Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1147325Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1147521Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1148023Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1148227Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1148436Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1148442Z 
2025-07-17T11:07:27.1148532Z SIGABRT(6), PID: 1828349, Thread 1828824: 
2025-07-17T11:07:27.1149069Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1149277Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1149482Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1149679Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1150172Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1150379Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1150650Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1150656Z 
2025-07-17T11:07:27.1150750Z SIGABRT(6), PID: 1828347, Thread 1833224: 
2025-07-17T11:07:27.1151286Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1151495Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1151697Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1151897Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1152277Z frame #4: <unknown function> + 0x5a02023 (0x7f1a2379b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1152848Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1a2379f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1153442Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1a237961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1153831Z frame #7: <unknown function> + 0x79d655 (0x7f1a34c86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1154130Z frame #8: <unknown function> + 0xdc253 (0x7f19ebcd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1154351Z frame #9: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1154568Z frame #10: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1154573Z 
2025-07-17T11:07:27.1154668Z SIGABRT(6), PID: 1828348, Thread 1828721: 
2025-07-17T11:07:27.1155147Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1155363Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1155568Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1155771Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1156282Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1156492Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1156702Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1156708Z 
2025-07-17T11:07:27.1156804Z SIGABRT(6), PID: 1828349, Thread 1828825: 
2025-07-17T11:07:27.1157280Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1157571Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1157774Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1157979Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1158475Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1158684Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1158951Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1158955Z 
2025-07-17T11:07:27.1159052Z SIGABRT(6), PID: 1828348, Thread 1828725: 
2025-07-17T11:07:27.1159591Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1159808Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1160011Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1160214Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1160713Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1160925Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1161134Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1161142Z 
2025-07-17T11:07:27.1161234Z SIGABRT(6), PID: 1828349, Thread 1828826: 
2025-07-17T11:07:27.1161708Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1161914Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1162123Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1162385Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1162885Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1163095Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1163307Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1163315Z 
2025-07-17T11:07:27.1163403Z SIGABRT(6), PID: 1828349, Thread 1828827: 
2025-07-17T11:07:27.1163878Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1164087Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1164295Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1164493Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1164994Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1165203Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1165489Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1165493Z 
2025-07-17T11:07:27.1165582Z SIGABRT(6), PID: 1828348, Thread 1828729: 
2025-07-17T11:07:27.1166053Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1166261Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1166464Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1166661Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1167228Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1167443Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1167738Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1167743Z 
2025-07-17T11:07:27.1167831Z SIGABRT(6), PID: 1828349, Thread 1828828: 
2025-07-17T11:07:27.1168304Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1168511Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1168716Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1168915Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1169413Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1169622Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1169831Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1169835Z 
2025-07-17T11:07:27.1169922Z SIGABRT(6), PID: 1828348, Thread 1828733: 
2025-07-17T11:07:27.1170465Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1170673Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1170878Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1171077Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1171576Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1171789Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1171994Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1171998Z 
2025-07-17T11:07:27.1172093Z SIGABRT(6), PID: 1828349, Thread 1828829: 
2025-07-17T11:07:27.1172561Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1172767Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1172969Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1173169Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1173734Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1173944Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1174151Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1174156Z 
2025-07-17T11:07:27.1174250Z SIGABRT(6), PID: 1828348, Thread 1828739: 
2025-07-17T11:07:27.1174722Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1174934Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1175202Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1175409Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1176024Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1176238Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1176447Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1176453Z 
2025-07-17T11:07:27.1176544Z SIGABRT(6), PID: 1828349, Thread 1828830: 
2025-07-17T11:07:27.1177016Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1177224Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1177426Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1177628Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1178123Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1178332Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1178622Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1178626Z 
2025-07-17T11:07:27.1178724Z SIGABRT(6), PID: 1828348, Thread 1828742: 
2025-07-17T11:07:27.1179193Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1179401Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1179607Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1179808Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1180304Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1180514Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1180723Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1180727Z 
2025-07-17T11:07:27.1180819Z SIGABRT(6), PID: 1828349, Thread 1828831: 
2025-07-17T11:07:27.1181289Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1181496Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1181792Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1181990Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1182488Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1182697Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1182909Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1182913Z 
2025-07-17T11:07:27.1183001Z SIGABRT(6), PID: 1828348, Thread 1828746: 
2025-07-17T11:07:27.1183543Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1183825Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1184031Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1184228Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1184720Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1184925Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1185131Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1185134Z 
2025-07-17T11:07:27.1185223Z SIGABRT(6), PID: 1828347, Thread 1833225: 
2025-07-17T11:07:27.1185698Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1185907Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1186114Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1186312Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1186757Z frame #4: <unknown function> + 0x5a02023 (0x7f1a2379b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1187326Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1a2379f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1187923Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1a237961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1188309Z frame #7: <unknown function> + 0x79d655 (0x7f1a34c86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1188546Z frame #8: <unknown function> + 0xdc253 (0x7f19ebcd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1188756Z frame #9: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1188983Z frame #10: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1188987Z 
2025-07-17T11:07:27.1189077Z SIGABRT(6), PID: 1828348, Thread 1828749: 
2025-07-17T11:07:27.1189548Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1189755Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1190030Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1190230Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1190731Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1190940Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1191151Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1191155Z 
2025-07-17T11:07:27.1191244Z SIGABRT(6), PID: 1828349, Thread 1828832: 
2025-07-17T11:07:27.1191779Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1191994Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1192261Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1192466Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1192959Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1193168Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1193375Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1193379Z 
2025-07-17T11:07:27.1193468Z SIGABRT(6), PID: 1828348, Thread 1828752: 
2025-07-17T11:07:27.1193933Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1194142Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1194342Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1194542Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1195100Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1195313Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1195521Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1195524Z 
2025-07-17T11:07:27.1195618Z SIGABRT(6), PID: 1828349, Thread 1828833: 
2025-07-17T11:07:27.1196082Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1196295Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1196498Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1196697Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1197195Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1197405Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1197626Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1197630Z 
2025-07-17T11:07:27.1197718Z SIGABRT(6), PID: 1828348, Thread 1828755: 
2025-07-17T11:07:27.1198257Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1198464Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1198669Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1198865Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1199363Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1199567Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1199857Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1199864Z 
2025-07-17T11:07:27.1200012Z SIGABRT(6), PID: 1828349, Thread 1828834: 
2025-07-17T11:07:27.1200483Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1200688Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1200893Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1201093Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1201584Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1201791Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1201999Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1202006Z 
2025-07-17T11:07:27.1202094Z SIGABRT(6), PID: 1828348, Thread 1828759: 
2025-07-17T11:07:27.1202563Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1202765Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1203034Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1203234Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1203727Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1203934Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1204153Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1204157Z 
2025-07-17T11:07:27.1204245Z SIGABRT(6), PID: 1828349, Thread 1828835: 
2025-07-17T11:07:27.1204710Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1204919Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1205122Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1205320Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1205816Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1206094Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1206303Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1206307Z 
2025-07-17T11:07:27.1206397Z SIGABRT(6), PID: 1828348, Thread 1828763: 
2025-07-17T11:07:27.1206860Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1207069Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1207271Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1207470Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1208023Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1208295Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1208502Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1208506Z 
2025-07-17T11:07:27.1208600Z SIGABRT(6), PID: 1828349, Thread 1828836: 
2025-07-17T11:07:27.1209065Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1209271Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1209475Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1209675Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1210163Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1210377Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1210583Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1210588Z 
2025-07-17T11:07:27.1210678Z SIGABRT(6), PID: 1828348, Thread 1828765: 
2025-07-17T11:07:27.1211208Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1211419Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1211623Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1211824Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1212320Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1212530Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1212736Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1212739Z 
2025-07-17T11:07:27.1212831Z SIGABRT(6), PID: 1828347, Thread 1833226: 
2025-07-17T11:07:27.1213299Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1213511Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1213715Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1213984Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1214366Z frame #4: <unknown function> + 0x5a02023 (0x7f1a2379b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1214933Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1a2379f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1215520Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1a237961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1215975Z frame #7: <unknown function> + 0x79d655 (0x7f1a34c86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1216291Z frame #8: <unknown function> + 0xdc253 (0x7f19ebcd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1216578Z frame #9: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1216799Z frame #10: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1216804Z 
2025-07-17T11:07:27.1216897Z SIGABRT(6), PID: 1828348, Thread 1828768: 
2025-07-17T11:07:27.1217370Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1217576Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1217782Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1217979Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1218487Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1218697Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1218907Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1218911Z 
2025-07-17T11:07:27.1219000Z SIGABRT(6), PID: 1828349, Thread 1828837: 
2025-07-17T11:07:27.1219551Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1219761Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1219968Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1220169Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1220668Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1220877Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1221086Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1221090Z 
2025-07-17T11:07:27.1221177Z SIGABRT(6), PID: 1828348, Thread 1828771: 
2025-07-17T11:07:27.1221645Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1221849Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1222059Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1222254Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1222833Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1223039Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1223248Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1223253Z 
2025-07-17T11:07:27.1223343Z SIGABRT(6), PID: 1828349, Thread 1828838: 
2025-07-17T11:07:27.1223809Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1224016Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1224288Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1224490Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1225046Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1225254Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1225466Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1225470Z 
2025-07-17T11:07:27.1225557Z SIGABRT(6), PID: 1828348, Thread 1828774: 
2025-07-17T11:07:27.1226027Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1226235Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1226436Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1226639Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1227135Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1227344Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1227621Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1227625Z 
2025-07-17T11:07:27.1227722Z SIGABRT(6), PID: 1828349, Thread 1828839: 
2025-07-17T11:07:27.1228192Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1228400Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1228605Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1228803Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1229296Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1229506Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1229711Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1229715Z 
2025-07-17T11:07:27.1229805Z SIGABRT(6), PID: 1828348, Thread 1828777: 
2025-07-17T11:07:27.1230272Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1230567Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1230770Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1230967Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1231459Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1231671Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1231878Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1231882Z 
2025-07-17T11:07:27.1231972Z SIGABRT(6), PID: 1828349, Thread 1828840: 
2025-07-17T11:07:27.1232500Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1232768Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1232973Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1233172Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1233662Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1233873Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1234080Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1234083Z 
2025-07-17T11:07:27.1234176Z SIGABRT(6), PID: 1828348, Thread 1828780: 
2025-07-17T11:07:27.1234643Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1234852Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1235056Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1235253Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1235803Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1236017Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1236226Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1236232Z 
2025-07-17T11:07:27.1236323Z SIGABRT(6), PID: 1828349, Thread 1828841: 
2025-07-17T11:07:27.1236796Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1237001Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1237206Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1237408Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1237908Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1238113Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1238323Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1238390Z 
2025-07-17T11:07:27.1238481Z SIGABRT(6), PID: 1828348, Thread 1828782: 
2025-07-17T11:07:27.1238951Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1239155Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1239362Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1239561Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1240054Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1240435Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1240652Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1240716Z 
2025-07-17T11:07:27.1240810Z SIGABRT(6), PID: 1828349, Thread 1828842: 
2025-07-17T11:07:27.1241281Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1241484Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1241690Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1241891Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1242388Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1242594Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1242807Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1242810Z 
2025-07-17T11:07:27.1242899Z SIGABRT(6), PID: 1828348, Thread 1828783: 
2025-07-17T11:07:27.1243375Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1243645Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1243855Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1244050Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1244547Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1244756Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1244967Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1244971Z 
2025-07-17T11:07:27.1245058Z SIGABRT(6), PID: 1828347, Thread 1833227: 
2025-07-17T11:07:27.1245532Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1245738Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1245943Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1246138Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1246515Z frame #4: <unknown function> + 0x5a02023 (0x7f1a2379b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1247152Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1a2379f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1247743Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1a237961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1248134Z frame #7: <unknown function> + 0x79d655 (0x7f1a34c86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1248366Z frame #8: <unknown function> + 0xdc253 (0x7f19ebcd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1248578Z frame #9: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1248854Z frame #10: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1248915Z 
2025-07-17T11:07:27.1249010Z SIGABRT(6), PID: 1828348, Thread 1828784: 
2025-07-17T11:07:27.1249482Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1249690Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1249893Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1250095Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1250593Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1250804Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1251014Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1251020Z 
2025-07-17T11:07:27.1251111Z SIGABRT(6), PID: 1828349, Thread 1828843: 
2025-07-17T11:07:27.1251572Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1251783Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1252055Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1252257Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1252750Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1252960Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1253169Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1253173Z 
2025-07-17T11:07:27.1253264Z SIGABRT(6), PID: 1828348, Thread 1828785: 
2025-07-17T11:07:27.1253724Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1253933Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1254133Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1254330Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1254825Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1255108Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1255315Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1255407Z SIGABRT(6), PID: 1828349, Thread 1828844: 
2025-07-17T11:07:27.1255875Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1256153Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1256356Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1256562Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1257160Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1257455Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1257666Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1257673Z 
2025-07-17T11:07:27.1257676Z 
2025-07-17T11:07:27.1257765Z SIGABRT(6), PID: 1828348, Thread 1828786: 
2025-07-17T11:07:27.1258251Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1258462Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1258671Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1258872Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1259379Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1259588Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1259799Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1259803Z 
2025-07-17T11:07:27.1259892Z SIGABRT(6), PID: 1828349, Thread 1828845: 
2025-07-17T11:07:27.1260465Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1260677Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1260886Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1261086Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1261587Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1261798Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1262009Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1262013Z 
2025-07-17T11:07:27.1262103Z SIGABRT(6), PID: 1828348, Thread 1828787: 
2025-07-17T11:07:27.1262574Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1262778Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1262982Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1263260Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1263759Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1263964Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1264173Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1264178Z 
2025-07-17T11:07:27.1264265Z SIGABRT(6), PID: 1828349, Thread 1828846: 
2025-07-17T11:07:27.1264738Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1265004Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1265212Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1265471Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1265965Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1266172Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1266383Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1266387Z 
2025-07-17T11:07:27.1266477Z SIGABRT(6), PID: 1828348, Thread 1828788: 
2025-07-17T11:07:27.1266950Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1267154Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1267365Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1267562Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1268058Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1268338Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1268548Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1268551Z 
2025-07-17T11:07:27.1268641Z SIGABRT(6), PID: 1828349, Thread 1828847: 
2025-07-17T11:07:27.1269112Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1269325Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1269529Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1269731Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1270224Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1270434Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1270640Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1270644Z 
2025-07-17T11:07:27.1270735Z SIGABRT(6), PID: 1828347, Thread 1833228: 
2025-07-17T11:07:27.1271204Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1271478Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1271682Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1271885Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1272261Z frame #4: <unknown function> + 0x5a02023 (0x7f1a2379b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1272830Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1a2379f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1273486Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1a237961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1273932Z frame #7: <unknown function> + 0x79d655 (0x7f1a34c86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1274168Z frame #8: <unknown function> + 0xdc253 (0x7f19ebcd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1274377Z frame #9: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1274594Z frame #10: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1274599Z 
2025-07-17T11:07:27.1274692Z SIGABRT(6), PID: 1828348, Thread 1828789: 
2025-07-17T11:07:27.1275164Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1275378Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1275581Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1275786Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1276283Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1276555Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1276766Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1276770Z 
2025-07-17T11:07:27.1276862Z SIGABRT(6), PID: 1828349, Thread 1828848: 
2025-07-17T11:07:27.1277335Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1277541Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1277757Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1277954Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1278450Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1278658Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1278869Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1278873Z 
2025-07-17T11:07:27.1278959Z SIGABRT(6), PID: 1828349, Thread 1828849: 
2025-07-17T11:07:27.1279425Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1279702Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1279912Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1280107Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1280602Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1280806Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1281017Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1281021Z 
2025-07-17T11:07:27.1281109Z SIGABRT(6), PID: 1828348, Thread 1828790: 
2025-07-17T11:07:27.1281643Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1281909Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1282116Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1282313Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1282807Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1283013Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1283224Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1283229Z 
2025-07-17T11:07:27.1283320Z SIGABRT(6), PID: 1828349, Thread 1828850: 
2025-07-17T11:07:27.1283788Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1283993Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1284197Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1284393Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1284952Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1285161Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1285372Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1285376Z 
2025-07-17T11:07:27.1285465Z SIGABRT(6), PID: 1828348, Thread 1828791: 
2025-07-17T11:07:27.1285939Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1286142Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1286349Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1286547Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1287040Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1287244Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1287456Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1287525Z 
2025-07-17T11:07:27.1287617Z SIGABRT(6), PID: 1828349, Thread 1828851: 
2025-07-17T11:07:27.1288084Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1288292Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1288495Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1288694Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1289190Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1289463Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1289676Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1289851Z SIGABRT(6), PID: 1828348, Thread 1828792: 
2025-07-17T11:07:27.1290316Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1290526Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1290729Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1290930Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1291428Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1291642Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1291850Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1291854Z 
2025-07-17T11:07:27.1291857Z 
2025-07-17T11:07:27.1291949Z SIGABRT(6), PID: 1828349, Thread 1828852: 
2025-07-17T11:07:27.1292415Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1292690Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1292898Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1293097Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1293590Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1293802Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1294011Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1294016Z 
2025-07-17T11:07:27.1294111Z SIGABRT(6), PID: 1828348, Thread 1828793: 
2025-07-17T11:07:27.1294581Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1294789Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1294992Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1295190Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1295683Z frame #4: <unknown function> + 0x3475ab (0x7fa08de075ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1296024Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1296233Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1296238Z 
2025-07-17T11:07:27.1296329Z SIGABRT(6), PID: 1828349, Thread 1828853: 
2025-07-17T11:07:27.1296799Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1297007Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1297207Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1297485Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1297981Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1298267Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1298476Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1298482Z 
2025-07-17T11:07:27.1298571Z SIGABRT(6), PID: 1828348, Thread 1829407: 
2025-07-17T11:07:27.1299040Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1299492Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f9f9f66508a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1299702Z frame #2: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1299903Z frame #3: pthread_kill + 0x12c (0x7fa090f719fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1300072Z frame #4: raise + 0x16 (0x7fa090f1d476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1300231Z frame #5: abort + 0xd3 (0x7fa090f037f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1300472Z frame #6: <unknown function> + 0x972cc (0x7f9f0d1ee2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1300780Z frame #7: <unknown function> + 0x95f47 (0x7f9f0d1ecf47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1301015Z frame #8: <unknown function> + 0x34b71 (0x7f9f0d18bb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1301226Z frame #9: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1301439Z frame #10: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1301445Z 
2025-07-17T11:07:27.1301536Z SIGABRT(6), PID: 1828349, Thread 1828854: 
2025-07-17T11:07:27.1302009Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1302223Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1302428Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1302630Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1303131Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1303337Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1303547Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1303628Z 
2025-07-17T11:07:27.1303724Z SIGABRT(6), PID: 1828348, Thread 1832732: 
2025-07-17T11:07:27.1304193Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1304399Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1304569Z frame #2: __poll + 0x4f (0x7fa090ff3bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1304985Z frame #3: <unknown function> + 0x489f (0x7fa090aff89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:27.1305217Z frame #4: <unknown function> + 0x1483bc (0x55f38b6b33bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1305464Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1305752Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1306052Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1306281Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1306525Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55f38b69ba65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1306760Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1307009Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f38b699c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1307233Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1307470Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1307695Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1307933Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1308157Z frame #16: <unknown function> + 0x147734 (0x55f38b6b2734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1308385Z frame #17: <unknown function> + 0x21ca82 (0x55f38b787a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1308608Z frame #18: <unknown function> + 0x21c9e4 (0x55f38b7879e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1308887Z frame #19: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1309103Z frame #20: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1309107Z 
2025-07-17T11:07:27.1309198Z SIGABRT(6), PID: 1828347, Thread 1833229: 
2025-07-17T11:07:27.1309677Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1309894Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1310103Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1310306Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1310688Z frame #4: <unknown function> + 0x5a02023 (0x7f1a2379b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1311250Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1a2379f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1311841Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1a237961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1312290Z frame #7: <unknown function> + 0x79d655 (0x7f1a34c86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1312522Z frame #8: <unknown function> + 0xdc253 (0x7f19ebcd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1312729Z frame #9: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1312946Z frame #10: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1312951Z 
2025-07-17T11:07:27.1313040Z SIGABRT(6), PID: 1828349, Thread 1828855: 
2025-07-17T11:07:27.1313515Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1313780Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1313992Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1314252Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1314760Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1314972Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1315184Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1315188Z 
2025-07-17T11:07:27.1315280Z SIGABRT(6), PID: 1828349, Thread 1828856: 
2025-07-17T11:07:27.1315755Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1315963Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1316173Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1316373Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1316867Z frame #4: <unknown function> + 0x3475ab (0x7f8d33b665ab in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-2f7c42d4.3.18.so)
2025-07-17T11:07:27.1317142Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1317356Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1317361Z 
2025-07-17T11:07:27.1317452Z SIGABRT(6), PID: 1828349, Thread 1829406: 
2025-07-17T11:07:27.1317927Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1318381Z frame #1: c10::FatalSignalHandler::fatalSignalHandler(int) + 0x27a (0x7f8c453c608a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1318587Z frame #2: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1318780Z frame #3: pthread_kill + 0x12c (0x7f8d36cd09fc in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1318948Z frame #4: raise + 0x16 (0x7f8d36c7c476 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1319113Z frame #5: abort + 0xd3 (0x7f8d36c627f3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1319348Z frame #6: <unknown function> + 0x972cc (0x7f8bb2f4f2cc in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1319581Z frame #7: <unknown function> + 0x95f47 (0x7f8bb2f4df47 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1319807Z frame #8: <unknown function> + 0x34b71 (0x7f8bb2eecb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1320018Z frame #9: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1320299Z frame #10: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1320303Z 
2025-07-17T11:07:27.1320395Z SIGABRT(6), PID: 1828349, Thread 1832934: 
2025-07-17T11:07:27.1320876Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1321088Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1321254Z frame #2: __poll + 0x4f (0x7f8d36d52bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1321661Z frame #3: <unknown function> + 0x489f (0x7f8d3685e89f in /opt/conda/envs/py_3.10/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so)
2025-07-17T11:07:27.1321969Z frame #4: <unknown function> + 0x1483bc (0x55d832bc43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1322223Z frame #5: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1322511Z frame #6: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1322751Z frame #7: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1322975Z frame #8: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1323221Z frame #9: _PyEval_EvalFrameDefault + 0x49b5 (0x55d832baca65 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1323450Z frame #10: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1323700Z frame #11: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d832baac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1323925Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1324169Z frame #13: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1324396Z frame #14: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1324634Z frame #15: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1324851Z frame #16: <unknown function> + 0x147734 (0x55d832bc3734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1325135Z frame #17: <unknown function> + 0x21ca82 (0x55d832c98a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1325358Z frame #18: <unknown function> + 0x21c9e4 (0x55d832c989e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1325574Z frame #19: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1325785Z frame #20: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1325789Z 
2025-07-17T11:07:27.1325884Z SIGABRT(6), PID: 1828348, Thread 1832733: 
2025-07-17T11:07:27.1326361Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1326574Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1326780Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1327005Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa090f6f2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1327450Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f9fcb3ad048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.1327888Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f9fcb3afbf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.1328189Z frame #6: <unknown function> + 0xdc253 (0x7f9f9d8d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1328405Z frame #7: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1328616Z frame #8: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1328620Z 
2025-07-17T11:07:27.1328711Z SIGABRT(6), PID: 1828348, Thread 1832734: 
2025-07-17T11:07:27.1329187Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1329397Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1329603Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1329888Z frame #3: pthread_cond_clockwait + 0x1fd (0x7fa090f6f2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1330367Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f9fcb3a7741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.1330666Z frame #5: <unknown function> + 0xdc253 (0x7f9f9d8d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1330872Z frame #6: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1331084Z frame #7: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1331088Z 
2025-07-17T11:07:27.1331177Z SIGABRT(6), PID: 1828349, Thread 1832935: 
2025-07-17T11:07:27.1331646Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1331855Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1332060Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1332287Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f8d36cce2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1332725Z frame #4: c10d::ProcessGroupNCCL::Watchdog::runLoop() + 0x118 (0x7f8c7110e048 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.1333209Z frame #5: c10d::ProcessGroupNCCL::Watchdog::run() + 0x117 (0x7f8c71110bf7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.1333438Z frame #6: <unknown function> + 0xdc253 (0x7f8c43632253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1333648Z frame #7: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1333855Z frame #8: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1333949Z SIGABRT(6), PID: 1828347, Thread 1833230: 
2025-07-17T11:07:27.1334414Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1334624Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1334829Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1335031Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1335409Z frame #4: <unknown function> + 0x5a02023 (0x7f1a2379b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1336041Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1a2379f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1336638Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1a237961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1337122Z frame #7: <unknown function> + 0x79d655 (0x7f1a34c86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1337357Z frame #8: <unknown function> + 0xdc253 (0x7f19ebcd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1337571Z frame #9: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1337784Z frame #10: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1337789Z 
2025-07-17T11:07:27.1337880Z SIGABRT(6), PID: 1828348, Thread 1832735: 
2025-07-17T11:07:27.1338441Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1338656Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1338899Z frame #2: ioctl + 0x3f (0x7fa090ff594f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1339139Z frame #3: <unknown function> + 0x13eef0 (0x7f9f0d295ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1339372Z frame #4: <unknown function> + 0x1378c4 (0x7f9f0d28e8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1339609Z frame #5: <unknown function> + 0x95b2b (0x7f9f0d1ecb2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1339837Z frame #6: <unknown function> + 0x34b71 (0x7f9f0d18bb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1340047Z frame #7: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1340256Z frame #8: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1340261Z 
2025-07-17T11:07:27.1340264Z 
2025-07-17T11:07:27.1340356Z SIGABRT(6), PID: 1828349, Thread 1832936: 
2025-07-17T11:07:27.1340832Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1341042Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1341245Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1341550Z frame #3: pthread_cond_clockwait + 0x1fd (0x7f8d36cce2dd in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1342029Z frame #4: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x261 (0x7f8c71108741 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so)
2025-07-17T11:07:27.1342257Z frame #5: <unknown function> + 0xdc253 (0x7f8c43632253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1342465Z frame #6: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1342679Z frame #7: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1342687Z 
2025-07-17T11:07:27.1342776Z SIGABRT(6), PID: 1828349, Thread 1832937: 
2025-07-17T11:07:27.1343244Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1343454Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1343618Z frame #2: ioctl + 0x3f (0x7f8d36d5494f in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1343856Z frame #3: <unknown function> + 0x13eef0 (0x7f8bb2ff6ef0 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1344088Z frame #4: <unknown function> + 0x1378c4 (0x7f8bb2fef8c4 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1344316Z frame #5: <unknown function> + 0x95b2b (0x7f8bb2f4db2b in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1344614Z frame #6: <unknown function> + 0x34b71 (0x7f8bb2eecb71 in /opt/rocm/lib/libhsa-runtime64.so.1)
2025-07-17T11:07:27.1344825Z frame #7: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1345036Z frame #8: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1345040Z 
2025-07-17T11:07:27.1345132Z SIGABRT(6), PID: 1828348, Thread 1833153: 
2025-07-17T11:07:27.1345602Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1345813Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1345977Z frame #2: __poll + 0x4f (0x7fa090ff3bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1346255Z frame #3: <unknown function> + 0xd4540 (0x7f9f55405540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.1346466Z frame #4: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1346740Z frame #5: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1346744Z 
2025-07-17T11:07:27.1346833Z SIGABRT(6), PID: 1828348, Thread 1833162: 
2025-07-17T11:07:27.1347305Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1347514Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1347679Z frame #2: __poll + 0x4f (0x7fa090ff3bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1347890Z frame #3: <unknown function> + 0xd65a2 (0x7f9f554075a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.1348100Z frame #4: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1348309Z frame #5: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1348317Z 
2025-07-17T11:07:27.1348410Z SIGABRT(6), PID: 1828349, Thread 1833157: 
2025-07-17T11:07:27.1348875Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1349082Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1349307Z frame #2: __poll + 0x4f (0x7f8d36d52bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1349525Z frame #3: <unknown function> + 0xd4540 (0x7f8bfb166540 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.1349731Z frame #4: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1349942Z frame #5: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1349946Z 
2025-07-17T11:07:27.1350035Z SIGABRT(6), PID: 1828349, Thread 1833164: 
2025-07-17T11:07:27.1350508Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1350714Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1350875Z frame #2: __poll + 0x4f (0x7f8d36d52bcf in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1351088Z frame #3: <unknown function> + 0xd65a2 (0x7f8bfb1685a2 in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.1351297Z frame #4: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1351503Z frame #5: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1351508Z 
2025-07-17T11:07:27.1351598Z SIGABRT(6), PID: 1828348, Thread 1833175: 
2025-07-17T11:07:27.1352067Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1352362Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1352567Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1352771Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1352988Z frame #4: <unknown function> + 0xd1e7f (0x7f9f55402e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.1353197Z frame #5: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1353401Z frame #6: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1353405Z 
2025-07-17T11:07:27.1353496Z SIGABRT(6), PID: 1828349, Thread 1833174: 
2025-07-17T11:07:27.1354029Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1354300Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1354504Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1354709Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1354922Z frame #4: <unknown function> + 0xd1e7f (0x7f8bfb163e7f in /opt/rocm-6.4.1/lib/librccl.so.1)
2025-07-17T11:07:27.1355131Z frame #5: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1355335Z frame #6: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1355339Z 
2025-07-17T11:07:27.1355432Z SIGABRT(6), PID: 1828348, Thread 1833176: 
2025-07-17T11:07:27.1355897Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1356108Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1356309Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1356513Z frame #3: <unknown function> + 0x9ca53 (0x7fa090f77a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1356829Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55f38b67fbc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1357064Z frame #5: <unknown function> + 0x1d314f (0x55f38b73e14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1357287Z frame #6: <unknown function> + 0x1d2f6b (0x55f38b73df6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1357513Z frame #7: <unknown function> + 0x6df36 (0x55f38b5d8f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1357759Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1358000Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1358239Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1358468Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1358706Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1358935Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1359168Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1359393Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1359628Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1359918Z frame #17: <unknown function> + 0x147734 (0x55f38b6b2734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1360138Z frame #18: <unknown function> + 0x21ca82 (0x55f38b787a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1360357Z frame #19: <unknown function> + 0x21c9e4 (0x55f38b7879e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1360570Z frame #20: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1360785Z frame #21: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1360790Z 
2025-07-17T11:07:27.1360880Z SIGABRT(6), PID: 1828349, Thread 1833183: 
2025-07-17T11:07:27.1361419Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1361633Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1361908Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1362112Z frame #3: <unknown function> + 0x9ca53 (0x7f8d36cd6a53 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1362365Z frame #4: PyThread_acquire_lock_timed + 0x165 (0x55d832b90bc5 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1362587Z frame #5: <unknown function> + 0x1d314f (0x55d832c4f14f in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1362813Z frame #6: <unknown function> + 0x1d2f6b (0x55d832c4ef6b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1363032Z frame #7: <unknown function> + 0x6df36 (0x55d832ae9f36 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1363276Z frame #8: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1363502Z frame #9: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1363744Z frame #10: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1363969Z frame #11: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1364206Z frame #12: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1364430Z frame #13: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1364731Z frame #14: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1364960Z frame #15: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1365197Z frame #16: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1365421Z frame #17: <unknown function> + 0x147734 (0x55d832bc3734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1365643Z frame #18: <unknown function> + 0x21ca82 (0x55d832c98a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1365868Z frame #19: <unknown function> + 0x21c9e4 (0x55d832c989e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1366081Z frame #20: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1366293Z frame #21: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1366297Z 
2025-07-17T11:07:27.1366390Z SIGABRT(6), PID: 1828348, Thread 1833185: 
2025-07-17T11:07:27.1366871Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1367085Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1367253Z frame #2: read + 0x4c (0x7fa090fef81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1367608Z frame #3: _Py_read + 0x4c (0x55f38b745a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1367834Z frame #4: <unknown function> + 0x208c3b (0x55f38b773c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1368057Z frame #5: <unknown function> + 0x14648d (0x55f38b6b148d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1368296Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55f38b6c0215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1368519Z frame #7: <unknown function> + 0x208ae7 (0x55f38b773ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1368733Z frame #8: <unknown function> + 0x208472 (0x55f38b773472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1368953Z frame #9: <unknown function> + 0x1da6ee (0x55f38b7456ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1369247Z frame #10: <unknown function> + 0x1483bc (0x55f38b6b33bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1369492Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1369780Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1370020Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55f38b6973bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1370236Z frame #14: <unknown function> + 0x147734 (0x55f38b6b2734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1370487Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55f38b699c55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1370712Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1370950Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1371175Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55f38b6a70ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1371414Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55f38b6977b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1371634Z frame #20: <unknown function> + 0x147734 (0x55f38b6b2734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1371852Z frame #21: <unknown function> + 0x21ca82 (0x55f38b787a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1372068Z frame #22: <unknown function> + 0x21c9e4 (0x55f38b7879e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1372346Z frame #23: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1372559Z frame #24: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1372571Z 
2025-07-17T11:07:27.1372661Z SIGABRT(6), PID: 1828347, Thread 1833231: 
2025-07-17T11:07:27.1373138Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f19eda64b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1373350Z frame #1: <unknown function> + 0x42520 (0x7f1adf31e520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1373562Z frame #2: <unknown function> + 0x91117 (0x7f1adf36d117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1373761Z frame #3: pthread_cond_wait + 0x211 (0x7f1adf36fa41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1374143Z frame #4: <unknown function> + 0x5a02023 (0x7f1a2379b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1374716Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f1a2379f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1375314Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f1a237961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1375697Z frame #7: <unknown function> + 0x79d655 (0x7f1a34c86655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1376085Z frame #8: <unknown function> + 0xdc253 (0x7f19ebcd1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1376298Z frame #9: <unknown function> + 0x94ac3 (0x7f1adf370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1376513Z frame #10: <unknown function> + 0x126850 (0x7f1adf402850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1376517Z 
2025-07-17T11:07:27.1376609Z SIGABRT(6), PID: 1828349, Thread 1833199: 
2025-07-17T11:07:27.1377082Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1377292Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1377538Z frame #2: read + 0x4c (0x7f8d36d4e81c in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1377720Z frame #3: _Py_read + 0x4c (0x55d832c56a6c in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1378023Z frame #4: <unknown function> + 0x208c3b (0x55d832c84c3b in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1378254Z frame #5: <unknown function> + 0x14648d (0x55d832bc248d in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1378503Z frame #6: PyObject_VectorcallMethod + 0x85 (0x55d832bd1215 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1378724Z frame #7: <unknown function> + 0x208ae7 (0x55d832c84ae7 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1378941Z frame #8: <unknown function> + 0x208472 (0x55d832c84472 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1379161Z frame #9: <unknown function> + 0x1da6ee (0x55d832c566ee in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1379388Z frame #10: <unknown function> + 0x1483bc (0x55d832bc43bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1379629Z frame #11: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1379861Z frame #12: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1380100Z frame #13: _PyEval_EvalFrameDefault + 0x30c (0x55d832ba83bc in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1380318Z frame #14: <unknown function> + 0x147734 (0x55d832bc3734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1380665Z frame #15: _PyEval_EvalFrameDefault + 0x2ba5 (0x55d832baac55 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1380899Z frame #16: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1381137Z frame #17: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1381365Z frame #18: _PyFunction_Vectorcall + 0x6c (0x55d832bb80ec in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1381599Z frame #19: _PyEval_EvalFrameDefault + 0x700 (0x55d832ba87b0 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1381822Z frame #20: <unknown function> + 0x147734 (0x55d832bc3734 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1382039Z frame #21: <unknown function> + 0x21ca82 (0x55d832c98a82 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1382262Z frame #22: <unknown function> + 0x21c9e4 (0x55d832c989e4 in /opt/conda/envs/py_3.10/bin/python)
2025-07-17T11:07:27.1382475Z frame #23: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1382687Z frame #24: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1382691Z 
2025-07-17T11:07:27.1382781Z SIGABRT(6), PID: 1828348, Thread 1833200: 
2025-07-17T11:07:27.1383261Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1383469Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1383766Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1383969Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1384350Z frame #4: <unknown function> + 0x5a02023 (0x7f9fd539b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1384921Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9fd539f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1385572Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9fd53961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1385957Z frame #7: <unknown function> + 0x79d655 (0x7f9fe6886655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1386255Z frame #8: <unknown function> + 0xdc253 (0x7f9f9d8d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1386464Z frame #9: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1386679Z frame #10: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1386684Z 
2025-07-17T11:07:27.1386776Z SIGABRT(6), PID: 1828349, Thread 1833256: 
2025-07-17T11:07:27.1387248Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1387458Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1387667Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1387868Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1388247Z frame #4: <unknown function> + 0x5a02023 (0x7f8c7b0fc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1388817Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8c7b1001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1389472Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8c7b0f71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1389861Z frame #7: <unknown function> + 0x79d655 (0x7f8c8c5e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1390092Z frame #8: <unknown function> + 0xdc253 (0x7f8c43632253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1390305Z frame #9: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1390519Z frame #10: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1390523Z 
2025-07-17T11:07:27.1390615Z SIGABRT(6), PID: 1828348, Thread 1833201: 
2025-07-17T11:07:27.1391087Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1391296Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1391497Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1391700Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1392075Z frame #4: <unknown function> + 0x5a02023 (0x7f9fd539b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1392711Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9fd539f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1393299Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9fd53961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1393683Z frame #7: <unknown function> + 0x79d655 (0x7f9fe6886655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1393910Z frame #8: <unknown function> + 0xdc253 (0x7f9f9d8d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1394179Z frame #9: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1394393Z frame #10: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1394457Z 
2025-07-17T11:07:27.1394549Z SIGABRT(6), PID: 1828349, Thread 1833257: 
2025-07-17T11:07:27.1395020Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1395232Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1395440Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1395643Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1396016Z frame #4: <unknown function> + 0x5a02023 (0x7f8c7b0fc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1396579Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8c7b1001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1397167Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8c7b0f71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1397552Z frame #7: <unknown function> + 0x79d655 (0x7f8c8c5e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1397844Z frame #8: <unknown function> + 0xdc253 (0x7f8c43632253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1398064Z frame #9: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1398286Z frame #10: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1398290Z 
2025-07-17T11:07:27.1398381Z SIGABRT(6), PID: 1828348, Thread 1833202: 
2025-07-17T11:07:27.1398856Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1399067Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1399271Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1399469Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1399845Z frame #4: <unknown function> + 0x5a02023 (0x7f9fd539b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1400405Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9fd539f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1400995Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9fd53961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1401445Z frame #7: <unknown function> + 0x79d655 (0x7f9fe6886655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1401674Z frame #8: <unknown function> + 0xdc253 (0x7f9f9d8d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1401880Z frame #9: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1402095Z frame #10: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1402099Z 
2025-07-17T11:07:27.1402186Z SIGABRT(6), PID: 1828349, Thread 1833258: 
2025-07-17T11:07:27.1402725Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1402937Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1403208Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1403407Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1403787Z frame #4: <unknown function> + 0x5a02023 (0x7f8c7b0fc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1404350Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8c7b1001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1404944Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8c7b0f71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1405327Z frame #7: <unknown function> + 0x79d655 (0x7f8c8c5e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1405558Z frame #8: <unknown function> + 0xdc253 (0x7f8c43632253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1405766Z frame #9: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1405982Z frame #10: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1405986Z 
2025-07-17T11:07:27.1406137Z SIGABRT(6), PID: 1828348, Thread 1833203: 
2025-07-17T11:07:27.1406615Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1406824Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1407030Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1407233Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1407604Z frame #4: <unknown function> + 0x5a02023 (0x7f9fd539b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1408170Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9fd539f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1408761Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9fd53961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1409141Z frame #7: <unknown function> + 0x79d655 (0x7f9fe6886655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1409370Z frame #8: <unknown function> + 0xdc253 (0x7f9f9d8d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1409647Z frame #9: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1409864Z frame #10: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1409868Z 
2025-07-17T11:07:27.1409959Z SIGABRT(6), PID: 1828349, Thread 1833259: 
2025-07-17T11:07:27.1410428Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1410638Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1410844Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1411048Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1411486Z frame #4: <unknown function> + 0x5a02023 (0x7f8c7b0fc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1412130Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8c7b1001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1412721Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8c7b0f71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1413107Z frame #7: <unknown function> + 0x79d655 (0x7f8c8c5e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1413331Z frame #8: <unknown function> + 0xdc253 (0x7f8c43632253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1413544Z frame #9: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1413754Z frame #10: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1413762Z 
2025-07-17T11:07:27.1413853Z SIGABRT(6), PID: 1828348, Thread 1833204: 
2025-07-17T11:07:27.1414321Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1414528Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1414792Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1414996Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1415367Z frame #4: <unknown function> + 0x5a02023 (0x7f9fd539b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1416006Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9fd539f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1416601Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9fd53961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1416980Z frame #7: <unknown function> + 0x79d655 (0x7f9fe6886655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1417211Z frame #8: <unknown function> + 0xdc253 (0x7f9f9d8d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1417416Z frame #9: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1417629Z frame #10: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1417633Z 
2025-07-17T11:07:27.1417721Z SIGABRT(6), PID: 1828349, Thread 1833260: 
2025-07-17T11:07:27.1418194Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1418497Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1418704Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1418905Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1419285Z frame #4: <unknown function> + 0x5a02023 (0x7f8c7b0fc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1419846Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8c7b1001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1420513Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8c7b0f71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1420966Z frame #7: <unknown function> + 0x79d655 (0x7f8c8c5e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1421193Z frame #8: <unknown function> + 0xdc253 (0x7f8c43632253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1421401Z frame #9: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1421615Z frame #10: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1421620Z 
2025-07-17T11:07:27.1421707Z SIGABRT(6), PID: 1828348, Thread 1833205: 
2025-07-17T11:07:27.1422182Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1422389Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1422597Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1422794Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1423169Z frame #4: <unknown function> + 0x5a02023 (0x7f9fd539b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1423807Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9fd539f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1424397Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9fd53961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1424781Z frame #7: <unknown function> + 0x79d655 (0x7f9fe6886655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1425010Z frame #8: <unknown function> + 0xdc253 (0x7f9f9d8d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1425217Z frame #9: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1425429Z frame #10: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1425433Z 
2025-07-17T11:07:27.1425526Z SIGABRT(6), PID: 1828349, Thread 1833261: 
2025-07-17T11:07:27.1425998Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1426206Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1426410Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1426610Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1427053Z frame #4: <unknown function> + 0x5a02023 (0x7f8c7b0fc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1427611Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8c7b1001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1428199Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8c7b0f71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1428586Z frame #7: <unknown function> + 0x79d655 (0x7f8c8c5e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1428867Z frame #8: <unknown function> + 0xdc253 (0x7f8c43632253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1429086Z frame #9: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1429358Z frame #10: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1429362Z 
2025-07-17T11:07:27.1429453Z SIGABRT(6), PID: 1828348, Thread 1833206: 
2025-07-17T11:07:27.1429928Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1430142Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1430347Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1430546Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1430914Z frame #4: <unknown function> + 0x5a02023 (0x7f9fd539b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1431478Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9fd539f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1432062Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9fd53961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1432508Z frame #7: <unknown function> + 0x79d655 (0x7f9fe6886655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1432742Z frame #8: <unknown function> + 0xdc253 (0x7f9f9d8d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1432954Z frame #9: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1433168Z frame #10: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1433178Z 
2025-07-17T11:07:27.1433266Z SIGABRT(6), PID: 1828349, Thread 1833262: 
2025-07-17T11:07:27.1433739Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1433945Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1434155Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1434354Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1434725Z frame #4: <unknown function> + 0x5a02023 (0x7f8c7b0fc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1435285Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8c7b1001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1435940Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8c7b0f71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1436322Z frame #7: <unknown function> + 0x79d655 (0x7f8c8c5e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1436550Z frame #8: <unknown function> + 0xdc253 (0x7f8c43632253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1436757Z frame #9: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1436971Z frame #10: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1436975Z 
2025-07-17T11:07:27.1437123Z SIGABRT(6), PID: 1828348, Thread 1833207: 
2025-07-17T11:07:27.1437597Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f9f9f664b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1437865Z frame #1: <unknown function> + 0x42520 (0x7fa090f1d520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1438079Z frame #2: <unknown function> + 0x91117 (0x7fa090f6c117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1438278Z frame #3: pthread_cond_wait + 0x211 (0x7fa090f6ea41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1438651Z frame #4: <unknown function> + 0x5a02023 (0x7f9fd539b023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1439213Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f9fd539f1ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1439800Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f9fd53961d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1440180Z frame #7: <unknown function> + 0x79d655 (0x7f9fe6886655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1440411Z frame #8: <unknown function> + 0xdc253 (0x7f9f9d8d1253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1440619Z frame #9: <unknown function> + 0x94ac3 (0x7fa090f6fac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1440894Z frame #10: <unknown function> + 0x126850 (0x7fa091001850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1440899Z 
2025-07-17T11:07:27.1440990Z SIGABRT(6), PID: 1828349, Thread 1833263: 
2025-07-17T11:07:27.1441468Z  frame #0: c10::FatalSignalHandler::stacktraceSignalHandler(bool) + 0x8a (0x7f8c453c5b9a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so)
2025-07-17T11:07:27.1441672Z frame #1: <unknown function> + 0x42520 (0x7f8d36c7c520 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1441881Z frame #2: <unknown function> + 0x91117 (0x7f8d36ccb117 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1442078Z frame #3: pthread_cond_wait + 0x211 (0x7f8d36ccda41 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1442452Z frame #4: <unknown function> + 0x5a02023 (0x7f8c7b0fc023 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1443017Z frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x8d (0x7f8c7b1001ed in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1443608Z frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x2a9 (0x7f8c7b0f71d9 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2025-07-17T11:07:27.1443989Z frame #7: <unknown function> + 0x79d655 (0x7f8c8c5e3655 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2025-07-17T11:07:27.1444283Z frame #8: <unknown function> + 0xdc253 (0x7f8c43632253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
2025-07-17T11:07:27.1444492Z frame #9: <unknown function> + 0x94ac3 (0x7f8d36cceac3 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1444708Z frame #10: <unknown function> + 0x126850 (0x7f8d36d60850 in /lib/x86_64-linux-gnu/libc.so.6)
2025-07-17T11:07:27.1444713Z 
2025-07-17T11:07:27.1444797Z FAILED [25.1908s] [100%]
2025-07-17T11:07:27.1444802Z 
2025-07-17T11:07:27.1444913Z ==================================== RERUNS ====================================
2025-07-17T11:07:27.1445098Z __________________ TestMultiProc.test_ddp_optimizer_cudagraph __________________
2025-07-17T11:07:27.1445194Z Traceback (most recent call last):
2025-07-17T11:07:27.1445669Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in wrapper
2025-07-17T11:07:27.1445761Z     self._join_processes(fn)
2025-07-17T11:07:27.1446229Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 920, in _join_processes
2025-07-17T11:07:27.1446339Z     self._check_return_codes(fn, elapsed_time)
2025-07-17T11:07:27.1446754Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 996, in _check_return_codes
2025-07-17T11:07:27.1446834Z     self.assertEqual(
2025-07-17T11:07:27.1447201Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4176, in assertEqual
2025-07-17T11:07:27.1447351Z     raise error_metas.pop()[0].to_error(  # type: ignore[index]
2025-07-17T11:07:27.1447449Z AssertionError: Scalars are not equal!
2025-07-17T11:07:27.1447456Z 
2025-07-17T11:07:27.1447542Z Expected 0 but got -6.
2025-07-17T11:07:27.1447623Z Absolute difference: 6
2025-07-17T11:07:27.1447708Z Relative difference: inf
2025-07-17T11:07:27.1447831Z Expected exit code 0 but got -6 for pid: 1817866
2025-07-17T11:07:27.1448019Z __________________ TestMultiProc.test_ddp_optimizer_cudagraph __________________
2025-07-17T11:07:27.1448110Z Traceback (most recent call last):
2025-07-17T11:07:27.1448509Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in wrapper
2025-07-17T11:07:27.1448595Z     self._join_processes(fn)
2025-07-17T11:07:27.1449066Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 920, in _join_processes
2025-07-17T11:07:27.1449175Z     self._check_return_codes(fn, elapsed_time)
2025-07-17T11:07:27.1449587Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 996, in _check_return_codes
2025-07-17T11:07:27.1449665Z     self.assertEqual(
2025-07-17T11:07:27.1450035Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4176, in assertEqual
2025-07-17T11:07:27.1450187Z     raise error_metas.pop()[0].to_error(  # type: ignore[index]
2025-07-17T11:07:27.1450284Z AssertionError: Scalars are not equal!
2025-07-17T11:07:27.1450288Z 
2025-07-17T11:07:27.1450369Z Expected 0 but got -6.
2025-07-17T11:07:27.1450448Z Absolute difference: 6
2025-07-17T11:07:27.1450530Z Relative difference: inf
2025-07-17T11:07:27.1450648Z Expected exit code 0 but got -6 for pid: 1822836
2025-07-17T11:07:27.1450759Z =================================== FAILURES ===================================
2025-07-17T11:07:27.1450940Z __________________ TestMultiProc.test_ddp_optimizer_cudagraph __________________
2025-07-17T11:07:27.1451027Z Traceback (most recent call last):
2025-07-17T11:07:27.1451399Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 656, in wrapper
2025-07-17T11:07:27.1451485Z     self._join_processes(fn)
2025-07-17T11:07:27.1451882Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 920, in _join_processes
2025-07-17T11:07:27.1452067Z     self._check_return_codes(fn, elapsed_time)
2025-07-17T11:07:27.1452478Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 996, in _check_return_codes
2025-07-17T11:07:27.1452555Z     self.assertEqual(
2025-07-17T11:07:27.1452918Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4176, in assertEqual
2025-07-17T11:07:27.1453059Z     raise error_metas.pop()[0].to_error(  # type: ignore[index]
2025-07-17T11:07:27.1453156Z AssertionError: Scalars are not equal!
2025-07-17T11:07:27.1453160Z 
2025-07-17T11:07:27.1453238Z Expected 0 but got -6.
2025-07-17T11:07:27.1453318Z Absolute difference: 6
2025-07-17T11:07:27.1453398Z Relative difference: inf
2025-07-17T11:07:27.1453572Z Expected exit code 0 but got -6 for pid: 1828342
2025-07-17T11:07:27.1454164Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-28cda1fd8ef32fac.xml -
2025-07-17T11:07:27.1454353Z =========================== short test summary info ============================
2025-07-17T11:07:27.1454766Z FAILED [25.1908s] distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph - AssertionError: Scalars are not equal!
2025-07-17T11:07:27.1454772Z 
2025-07-17T11:07:27.1454854Z Expected 0 but got -6.
2025-07-17T11:07:27.1454931Z Absolute difference: 6
2025-07-17T11:07:27.1455010Z Relative difference: inf
2025-07-17T11:07:27.1455122Z Expected exit code 0 but got -6 for pid: 1828342
2025-07-17T11:07:27.1455248Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2025-07-17T11:07:27.1455399Z ============= 1 failed, 22 deselected, 2 rerun in 76.42s (0:01:16) =============
2025-07-17T11:07:27.1456010Z /opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 24 leaked semaphore objects to clean up at shutdown
2025-07-17T11:07:27.1456162Z   warnings.warn('resource_tracker: There appear to be %d '
2025-07-17T11:07:27.1456236Z Got exit code 1
2025-07-17T11:07:27.1456531Z Test failed consistently, continuing with the rest of the tests due to continue-through-error being set
2025-07-17T11:07:27.1457613Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1457701Z   import pkg_resources
2025-07-17T11:07:27.1458173Z Test results will be stored in test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-01bf0091d99ba259.xml
2025-07-17T11:07:27.1458295Z ============================= test session starts ==============================
2025-07-17T11:07:27.1458544Z platform linux -- Python 3.10.18, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python
2025-07-17T11:07:27.1458630Z cachedir: .pytest_cache
2025-07-17T11:07:27.1458987Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2025-07-17T11:07:27.1459081Z rootdir: /var/lib/jenkins/pytorch
2025-07-17T11:07:27.1459160Z configfile: pytest.ini
2025-07-17T11:07:27.1459527Z plugins: xdist-3.3.1, rerunfailures-14.0, xdoctest-1.1.0, flakefinder-1.1.0, cpp-2.3.0, subtests-0.13.1, hypothesis-5.35.1, typeguard-4.3.0
2025-07-17T11:07:27.1459680Z collecting ... collected 60 items / 9 deselected / 51 selected
2025-07-17T11:07:27.1459783Z stepcurrent: skipping 9 already run items.
2025-07-17T11:07:27.1459869Z Running 14 items in this shard
2025-07-17T11:07:27.1459875Z 
2025-07-17T11:07:27.1460551Z distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing I0717 11:03:58.300000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1834914
2025-07-17T11:07:27.1460991Z I0717 11:03:58.301000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1834915
2025-07-17T11:07:27.1461341Z I0717 11:03:58.302000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1834916
2025-07-17T11:07:27.1461689Z I0717 11:03:58.302000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1834917
2025-07-17T11:07:27.1462034Z I0717 11:03:58.303000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1834918
2025-07-17T11:07:27.1462453Z I0717 11:03:58.304000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1834919
2025-07-17T11:07:27.1462800Z I0717 11:03:58.306000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1834920
2025-07-17T11:07:27.1463216Z I0717 11:03:58.307000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1834921
2025-07-17T11:07:27.1464217Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1464298Z   import pkg_resources
2025-07-17T11:07:27.1465279Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1465360Z   import pkg_resources
2025-07-17T11:07:27.1466334Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1466413Z   import pkg_resources
2025-07-17T11:07:27.1467443Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1467526Z   import pkg_resources
2025-07-17T11:07:27.1468497Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1468587Z   import pkg_resources
2025-07-17T11:07:27.1469560Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1469639Z   import pkg_resources
2025-07-17T11:07:27.1470608Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1470749Z   import pkg_resources
2025-07-17T11:07:27.1471725Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1471801Z   import pkg_resources
2025-07-17T11:07:27.1472289Z [rank7]:W0717 11:04:20.448000 1834921 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1472762Z [rank0]:W0717 11:04:20.452000 1834914 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1473289Z [rank3]:W0717 11:04:20.453000 1834917 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1473816Z [rank4]:W0717 11:04:20.455000 1834918 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1474281Z [rank6]:W0717 11:04:20.456000 1834920 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1474743Z [rank1]:W0717 11:04:20.462000 1834915 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1475208Z [rank5]:W0717 11:04:20.466000 1834919 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1475673Z [rank2]:W0717 11:04:20.471000 1834916 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1476524Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1476600Z   warnings.warn(
2025-07-17T11:07:27.1477501Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1477579Z   warnings.warn(
2025-07-17T11:07:27.1478425Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1478506Z   warnings.warn(
2025-07-17T11:07:27.1479336Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1479411Z   warnings.warn(
2025-07-17T11:07:27.1480240Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1480313Z   warnings.warn(
2025-07-17T11:07:27.1481146Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1481308Z   warnings.warn(
2025-07-17T11:07:27.1482165Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1482242Z   warnings.warn(
2025-07-17T11:07:27.1483072Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1483149Z   warnings.warn(
2025-07-17T11:07:27.1483228Z PASSED [30.3546s] [  7%]
2025-07-17T11:07:27.1483902Z distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor I0717 11:04:28.658000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1840445
2025-07-17T11:07:27.1484327Z I0717 11:04:28.659000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1840446
2025-07-17T11:07:27.1489687Z I0717 11:04:28.661000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1840447
2025-07-17T11:07:27.1490104Z I0717 11:04:28.662000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1840448
2025-07-17T11:07:27.1490475Z I0717 11:04:28.663000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1840449
2025-07-17T11:07:27.1490817Z I0717 11:04:28.664000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1840450
2025-07-17T11:07:27.1491160Z I0717 11:04:28.665000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1840451
2025-07-17T11:07:27.1491502Z I0717 11:04:28.666000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1840452
2025-07-17T11:07:27.1492510Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1492704Z   import pkg_resources
2025-07-17T11:07:27.1493685Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1493766Z   import pkg_resources
2025-07-17T11:07:27.1494730Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1494815Z   import pkg_resources
2025-07-17T11:07:27.1495775Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1495852Z   import pkg_resources
2025-07-17T11:07:27.1496897Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1497067Z   import pkg_resources
2025-07-17T11:07:27.1498028Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1498105Z   import pkg_resources
2025-07-17T11:07:27.1499082Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1499234Z   import pkg_resources
2025-07-17T11:07:27.1500193Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1500347Z   import pkg_resources
2025-07-17T11:07:27.1500827Z [rank7]:W0717 11:04:46.684000 1840452 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1501300Z [rank1]:W0717 11:04:46.956000 1840446 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1501760Z [rank2]:W0717 11:04:47.772000 1840447 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1502224Z [rank5]:W0717 11:04:48.000000 1840450 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1502687Z [rank6]:W0717 11:04:49.577000 1840451 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1503139Z [rank3]:W0717 11:04:49.826000 1840448 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1503679Z [rank4]:W0717 11:04:49.886000 1840449 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1504144Z [rank0]:W0717 11:04:49.897000 1840445 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1505006Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1505084Z   warnings.warn(
2025-07-17T11:07:27.1505926Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1505998Z   warnings.warn(
2025-07-17T11:07:27.1506826Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1506899Z   warnings.warn(
2025-07-17T11:07:27.1507727Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1507866Z   warnings.warn(
2025-07-17T11:07:27.1508707Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1508776Z   warnings.warn(
2025-07-17T11:07:27.1509664Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1509737Z   warnings.warn(
2025-07-17T11:07:27.1510570Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1510707Z   warnings.warn(
2025-07-17T11:07:27.1511534Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1511606Z   warnings.warn(
2025-07-17T11:07:27.1511686Z PASSED [31.5547s] [ 14%]
2025-07-17T11:07:27.1512302Z distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr I0717 11:05:00.215000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1846762
2025-07-17T11:07:27.1512655Z I0717 11:05:00.216000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1846763
2025-07-17T11:07:27.1513005Z I0717 11:05:00.217000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1846764
2025-07-17T11:07:27.1513345Z I0717 11:05:00.218000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1846765
2025-07-17T11:07:27.1513861Z I0717 11:05:00.219000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1846766
2025-07-17T11:07:27.1514205Z I0717 11:05:00.220000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1846767
2025-07-17T11:07:27.1514546Z I0717 11:05:00.221000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1846768
2025-07-17T11:07:27.1514885Z I0717 11:05:00.222000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1846769
2025-07-17T11:07:27.1515872Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1515949Z   import pkg_resources
2025-07-17T11:07:27.1516916Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1516991Z   import pkg_resources
2025-07-17T11:07:27.1517964Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1518108Z   import pkg_resources
2025-07-17T11:07:27.1519086Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1519161Z   import pkg_resources
2025-07-17T11:07:27.1520176Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1520257Z   import pkg_resources
2025-07-17T11:07:27.1521219Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1521357Z   import pkg_resources
2025-07-17T11:07:27.1522316Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1522396Z   import pkg_resources
2025-07-17T11:07:27.1523349Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1523428Z   import pkg_resources
2025-07-17T11:07:27.1523901Z [rank2]:W0717 11:05:19.343000 1846764 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1524451Z [rank6]:W0717 11:05:19.977000 1846768 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1524914Z [rank7]:W0717 11:05:19.987000 1846769 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1525375Z [rank5]:W0717 11:05:20.505000 1846767 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1525840Z [rank4]:W0717 11:05:20.766000 1846766 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1526305Z [rank3]:W0717 11:05:21.560000 1846765 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1526762Z [rank1]:W0717 11:05:24.289000 1846763 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1527224Z [rank0]:W0717 11:05:24.300000 1846762 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1527304Z PASSED [30.3525s] [ 21%]
2025-07-17T11:07:27.1527925Z distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager I0717 11:05:30.569000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1851125
2025-07-17T11:07:27.1528342Z I0717 11:05:30.570000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1851126
2025-07-17T11:07:27.1528679Z I0717 11:05:30.571000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1851127
2025-07-17T11:07:27.1529024Z I0717 11:05:30.572000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1851128
2025-07-17T11:07:27.1529362Z I0717 11:05:30.573000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1851129
2025-07-17T11:07:27.1529703Z I0717 11:05:30.574000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1851130
2025-07-17T11:07:27.1530105Z I0717 11:05:30.575000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1851131
2025-07-17T11:07:27.1530505Z I0717 11:05:30.576000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1851132
2025-07-17T11:07:27.1531489Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1531568Z   import pkg_resources
2025-07-17T11:07:27.1532534Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1532611Z   import pkg_resources
2025-07-17T11:07:27.1533591Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1533675Z   import pkg_resources
2025-07-17T11:07:27.1534729Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1534811Z   import pkg_resources
2025-07-17T11:07:27.1535780Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1535861Z   import pkg_resources
2025-07-17T11:07:27.1536886Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1536963Z   import pkg_resources
2025-07-17T11:07:27.1537925Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1538001Z   import pkg_resources
2025-07-17T11:07:27.1538968Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1539129Z   import pkg_resources
2025-07-17T11:07:27.1539605Z [rank0]:W0717 11:05:58.625000 1851125 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1540068Z [rank7]:W0717 11:05:58.637000 1851132 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1540601Z [rank3]:W0717 11:05:58.656000 1851128 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1541065Z [rank1]:W0717 11:05:58.657000 1851126 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1541603Z [rank5]:W0717 11:05:58.660000 1851130 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1542066Z [rank6]:W0717 11:05:58.661000 1851131 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1542529Z [rank4]:W0717 11:05:58.661000 1851129 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1542986Z [rank2]:W0717 11:05:58.664000 1851127 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1543062Z PASSED [39.9646s] [ 28%]
2025-07-17T11:07:27.1543480Z distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp SKIPPED [0.0003s] (Inaccurate results with fused SDPA kernels) [ 35%]
2025-07-17T11:07:27.1544164Z distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing I0717 11:06:10.536000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 0 with pid 1855560
2025-07-17T11:07:27.1544585Z I0717 11:06:10.537000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 1 with pid 1855561
2025-07-17T11:07:27.1544932Z I0717 11:06:10.538000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 2 with pid 1855562
2025-07-17T11:07:27.1545271Z I0717 11:06:10.539000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 3 with pid 1855563
2025-07-17T11:07:27.1545609Z I0717 11:06:10.540000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 4 with pid 1855564
2025-07-17T11:07:27.1545956Z I0717 11:06:10.541000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 5 with pid 1855565
2025-07-17T11:07:27.1546292Z I0717 11:06:10.542000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 6 with pid 1855566
2025-07-17T11:07:27.1546632Z I0717 11:06:10.543000 1834384 site-packages/torch/testing/_internal/common_distributed.py:729] Started process 7 with pid 1855567
2025-07-17T11:07:27.1547605Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1547683Z   import pkg_resources
2025-07-17T11:07:27.1548646Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1548804Z   import pkg_resources
2025-07-17T11:07:27.1549766Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1549842Z   import pkg_resources
2025-07-17T11:07:27.1550860Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1551003Z   import pkg_resources
2025-07-17T11:07:27.1551971Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1552045Z   import pkg_resources
2025-07-17T11:07:27.1553015Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1553089Z   import pkg_resources
2025-07-17T11:07:27.1554052Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1554130Z   import pkg_resources
2025-07-17T11:07:27.1555160Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:07:27.1555237Z   import pkg_resources
2025-07-17T11:07:27.1555708Z [rank5]:W0717 11:06:35.704000 1855565 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1556171Z [rank4]:W0717 11:06:35.705000 1855564 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1556637Z [rank2]:W0717 11:06:35.706000 1855562 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1557099Z [rank1]:W0717 11:06:35.706000 1855561 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1557564Z [rank7]:W0717 11:06:35.707000 1855567 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1558023Z [rank6]:W0717 11:06:35.707000 1855566 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1558494Z [rank0]:W0717 11:06:35.709000 1855560 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1559022Z [rank3]:W0717 11:06:35.709000 1855563 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1559866Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1559942Z   warnings.warn(
2025-07-17T11:07:27.1560836Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1560913Z   warnings.warn(
2025-07-17T11:07:27.1561743Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1561897Z   warnings.warn(
2025-07-17T11:07:27.1562719Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1562792Z   warnings.warn(
2025-07-17T11:07:27.1563614Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1563686Z   warnings.warn(
2025-07-17T11:07:27.1564510Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1564584Z   warnings.warn(
2025-07-17T11:07:27.1565471Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1565546Z   warnings.warn(
2025-07-17T11:07:27.1566371Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:304: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
2025-07-17T11:07:27.1566444Z   warnings.warn(
2025-07-17T11:07:27.1566714Z Running hf_bert_activation_checkpointing test for FSDP with recursive wrapping BertLayer instances
2025-07-17T11:07:27.1566971Z Running hf_bert_activation_checkpointing test for FSDP with recursive wrapping BertLayer instances
2025-07-17T11:07:27.1567222Z Running hf_bert_activation_checkpointing test for FSDP with recursive wrapping BertLayer instances
2025-07-17T11:07:27.1567474Z Running hf_bert_activation_checkpointing test for FSDP with recursive wrapping BertLayer instances
2025-07-17T11:07:27.1567726Z Running hf_bert_activation_checkpointing test for FSDP with recursive wrapping BertLayer instances
2025-07-17T11:07:27.1567978Z Running hf_bert_activation_checkpointing test for FSDP with recursive wrapping BertLayer instances
2025-07-17T11:07:27.1568226Z Running hf_bert_activation_checkpointing test for FSDP with recursive wrapping BertLayer instances
2025-07-17T11:07:27.1568478Z Running hf_bert_activation_checkpointing test for FSDP with recursive wrapping BertLayer instances
2025-07-17T11:07:27.1568624Z PASSED [61.4984s] [ 42%]
2025-07-17T11:07:27.1569358Z distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd [rank0]:W0717 11:07:14.107000 1834384 site-packages/torch/_logging/_internal.py:1156] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-07-17T11:07:27.1569435Z PASSED [3.1794s] [ 50%]
2025-07-17T11:07:27.1569756Z distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize PASSED [1.0553s] [ 57%]
2025-07-17T11:07:27.1570087Z distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_local_ddp PASSED [2.5335s] [ 64%]
2025-07-17T11:07:27.1570354Z distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer PASSED [0.2296s] [ 71%]
2025-07-17T11:07:27.1570713Z distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager PASSED [0.6593s] [ 78%]
2025-07-17T11:07:27.1571007Z distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor PASSED [0.0100s] [ 85%]
2025-07-17T11:07:27.1571359Z distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod PASSED [0.7743s] [ 92%]
2025-07-17T11:07:27.1571749Z distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training PASSED [2.0979s] [100%]
2025-07-17T11:07:27.1571757Z 
2025-07-17T11:07:27.1572344Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-01bf0091d99ba259.xml -
2025-07-17T11:07:27.1572510Z =========== 13 passed, 1 skipped, 9 deselected in 204.74s (0:03:24) ============
2025-07-17T11:07:27.1572905Z The following tests failed consistently: ['test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph']
2025-07-17T11:07:27.1572910Z 
2025-07-17T11:07:27.1573357Z FINISHED PRINTING LOG FILE of distributed/test_dynamo_distributed 2/2 (test/test-reports/distributed.test_dynamo_distributed_2.2_492ff260544723ef_.log)
2025-07-17T11:07:27.1573364Z 
2025-07-17T11:07:27.1573569Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-17T11:07:27.1573660Z Uploading artifacts took 0.00 seconds
2025-07-17T11:07:27.1573778Z distributed/test_dynamo_distributed 2/2 failed!
2025-07-17T11:07:27.1573986Z Running distributed/test_inductor_collectives 1/1 ... [2025-07-17 11:07:25.415937]
2025-07-17T11:07:27.1574146Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-17T11:07:27.1574885Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-17 11:07:25.416263]
2025-07-17T11:11:53.6238857Z 
2025-07-17T11:11:53.6240269Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_b7680240b74d3b29_.log
2025-07-17T11:11:53.6266115Z Running 57 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_recompute_is_always_banned_override_with_ac_False, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_recompute_is_always_banned_override_with_ac_True, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_async_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_all_gather_bucket, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared_graph_partition, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reduce_scatter_bucket, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reorder_peak_memory, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reorder_peak_memory_bucketed, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reorder_respects_wait_dep, test/distributed/test_inductor_collectives.py::TestSyncDecisionCrossRanks::test_sync_decision_cross_ranks
2025-07-17T11:11:53.6287804Z 
2025-07-17T11:11:57.3553240Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:11:57.3555422Z   import pkg_resources
2025-07-17T11:11:57.3695461Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:11:57.3697658Z   import pkg_resources
2025-07-17T11:11:57.4205596Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:11:57.4207662Z   import pkg_resources
2025-07-17T11:11:57.4209585Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:11:57.4210825Z   import pkg_resources
2025-07-17T11:11:57.4362297Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:11:57.4364348Z   import pkg_resources
2025-07-17T11:11:57.4374434Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:11:57.4375665Z   import pkg_resources
2025-07-17T11:11:57.4476110Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:11:57.4478371Z   import pkg_resources
2025-07-17T11:11:57.4480658Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-17T11:11:57.4481815Z   import pkg_resources
2025-07-17T11:11:58.8038564Z Running test batch 'tests to run' cost 1579.19 seconds
2025-07-17T11:11:58.8039193Z Emiting td_test_failure_stats_v2
2025-07-17T11:11:58.8041509Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1752750718_db3efcc262fe11f08e997efd5929c720
2025-07-17T11:12:00.9190198Z /var/lib/jenkins/pytorch/tools/stats/upload_metrics.py:156: UserWarning: Error uploading metric td_test_failure_stats_v2 to DynamoDB: Unable to locate credentials
2025-07-17T11:12:00.9192038Z   warn(f"Error uploading metric {metric_name} to DynamoDB: {e}")
2025-07-17T11:12:00.9192706Z distributed/test_dynamo_distributed 2/2 failed!
2025-07-17T11:12:01.5946394Z + sccache_epilogue
2025-07-17T11:12:01.5946874Z + echo '::group::Sccache Compilation Log'
2025-07-17T11:12:01.5947786Z ##[group]Sccache Compilation Log
2025-07-17T11:12:01.5948382Z + echo '=================== sccache compilation log ==================='
2025-07-17T11:12:01.5948909Z =================== sccache compilation log ===================
2025-07-17T11:12:01.5949570Z + python /var/lib/jenkins/pytorch/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log
2025-07-17T11:12:01.6103375Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
2025-07-17T11:12:01.6104354Z =========== If your build fails, please take a look at the log above for possible reasons ===========
2025-07-17T11:12:01.6145574Z + sccache --show-stats
2025-07-17T11:12:01.6146006Z Compile requests                   9183
2025-07-17T11:12:01.6146468Z Compile requests executed           573
2025-07-17T11:12:01.6146913Z Cache hits                           20
2025-07-17T11:12:01.6147336Z Cache hits (C/C++)                   20
2025-07-17T11:12:01.6147763Z Cache misses                        553
2025-07-17T11:12:01.6148198Z Cache misses (C/C++)                547
2025-07-17T11:12:01.6149003Z Cache misses (HIP)                    6
2025-07-17T11:12:01.6149496Z Cache hits rate                    3.49 %
2025-07-17T11:12:01.6149958Z Cache hits rate (C/C++)            3.53 %
2025-07-17T11:12:01.6150407Z Cache hits rate (HIP)              0.00 %
2025-07-17T11:12:01.6150848Z Cache timeouts                        0
2025-07-17T11:12:01.6151278Z Cache read errors                     0
2025-07-17T11:12:01.6151703Z Forced recaches                       0
2025-07-17T11:12:01.6152133Z Cache write errors                    0
2025-07-17T11:12:01.6152560Z Cache errors                          0
2025-07-17T11:12:01.6152979Z Compilations                        553
2025-07-17T11:12:01.6153406Z Compilation failures                  0
2025-07-17T11:12:01.6153850Z Non-cacheable compilations            0
2025-07-17T11:12:01.6154296Z Non-cacheable calls                 307
2025-07-17T11:12:01.6154726Z Non-compilation calls              8303
2025-07-17T11:12:01.6155172Z Unsupported compiler calls            0
2025-07-17T11:12:01.6155623Z Average cache write               0.000 s
2025-07-17T11:12:01.6156072Z Average compiler                  2.638 s
2025-07-17T11:12:01.6156516Z Average cache read hit            0.000 s
2025-07-17T11:12:01.6156983Z Failed distributed compilations       0
2025-07-17T11:12:01.6157296Z 
2025-07-17T11:12:01.6157451Z Non-cacheable reasons:
2025-07-17T11:12:01.6157849Z unknown source language             259
2025-07-17T11:12:01.6158287Z -E                                   48
2025-07-17T11:12:01.6158776Z 
2025-07-17T11:12:01.6159025Z Cache location                  Local disk: "/var/lib/jenkins/.cache/sccache"
2025-07-17T11:12:01.6159394Z Use direct/preprocessor mode?   yes
2025-07-17T11:12:01.6159672Z Version (client)                0.10.0
2025-07-17T11:12:01.6159940Z Cache size                           43 MiB
2025-07-17T11:12:01.6160213Z Max cache size                       10 GiB
2025-07-17T11:12:01.6160501Z + sccache --stop-server
2025-07-17T11:12:01.6190669Z Stopping sccache server...
2025-07-17T11:12:01.6194062Z Compile requests                   9183
2025-07-17T11:12:01.6194535Z Compile requests executed           573
2025-07-17T11:12:01.6194999Z Cache hits                           20
2025-07-17T11:12:01.6195419Z Cache hits (C/C++)                   20
2025-07-17T11:12:01.6195832Z Cache misses                        553
2025-07-17T11:12:01.6196434Z Cache misses (C/C++)                547
2025-07-17T11:12:01.6196865Z Cache misses (HIP)                    6
2025-07-17T11:12:01.6197487Z Cache hits rate                    3.49 %
2025-07-17T11:12:01.6197946Z Cache hits rate (C/C++)            3.53 %
2025-07-17T11:12:01.6198405Z Cache hits rate (HIP)              0.00 %
2025-07-17T11:12:01.6198846Z Cache timeouts                        0
2025-07-17T11:12:01.6199185Z Cache read errors                     0
2025-07-17T11:12:01.6199437Z Forced recaches                       0
2025-07-17T11:12:01.6199682Z Cache write errors                    0
2025-07-17T11:12:01.6199936Z Cache errors                          0
2025-07-17T11:12:01.6200186Z Compilations                        553
2025-07-17T11:12:01.6200446Z Compilation failures                  0
2025-07-17T11:12:01.6200713Z Non-cacheable compilations            0
2025-07-17T11:12:01.6200977Z Non-cacheable calls                 307
2025-07-17T11:12:01.6201234Z Non-compilation calls              8303
2025-07-17T11:12:01.6201499Z Unsupported compiler calls            0
2025-07-17T11:12:01.6201767Z Average cache write               0.000 s
2025-07-17T11:12:01.6202056Z Average compiler                  2.638 s
2025-07-17T11:12:01.6202316Z Average cache read hit            0.000 s
2025-07-17T11:12:01.6202585Z Failed distributed compilations       0
2025-07-17T11:12:01.6202765Z 
2025-07-17T11:12:01.6202848Z Non-cacheable reasons:
2025-07-17T11:12:01.6203069Z unknown source language             259
2025-07-17T11:12:01.6203318Z -E                                   48
2025-07-17T11:12:01.6203487Z 
2025-07-17T11:12:01.6203754Z Cache location                  Local disk: "/var/lib/jenkins/.cache/sccache"
2025-07-17T11:12:01.6204127Z Use direct/preprocessor mode?   yes
2025-07-17T11:12:01.6204405Z Version (client)                0.10.0
2025-07-17T11:12:01.6204666Z Cache size                           43 MiB
2025-07-17T11:12:01.6204932Z Max cache size                       10 GiB
2025-07-17T11:12:01.6205195Z + echo ::endgroup::
2025-07-17T11:12:01.6205552Z ##[endgroup]
2025-07-17T11:12:01.6285520Z ##[error]Process completed with exit code 1.
2025-07-17T11:12:01.6319412Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-07-17T11:12:01.6320168Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-07-17T11:12:01.6321043Z docker exec -t "16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test"
2025-07-17T11:12:01.6357138Z shell: /usr/bin/bash -e {0}
2025-07-17T11:12:01.6357389Z env:
2025-07-17T11:12:01.6357587Z   GIT_DEFAULT_BRANCH: main
2025-07-17T11:12:01.6357971Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T11:12:01.6358538Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T11:12:01.6359057Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T11:12:01.6359945Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T11:12:01.6360844Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T11:12:01.6361089Z   AWS_REGION: us-east-1
2025-07-17T11:12:01.6361394Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T11:12:01.6361713Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T11:12:01.6367117Z   AWS_SESSION_TOKEN: ***
2025-07-17T11:12:01.6367493Z   CONTAINER_NAME: 16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a
2025-07-17T11:12:01.6367905Z ##[endgroup]
2025-07-17T11:12:01.8510762Z ##[group]Run cat test/**/*_toprint.log || true
2025-07-17T11:12:01.8511135Z cat test/**/*_toprint.log || true
2025-07-17T11:12:01.8539177Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-17T11:12:01.8539513Z env:
2025-07-17T11:12:01.8539720Z   GIT_DEFAULT_BRANCH: main
2025-07-17T11:12:01.8540103Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T11:12:01.8540797Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T11:12:01.8541303Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T11:12:01.8542144Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T11:12:01.8542894Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T11:12:01.8543144Z   AWS_REGION: us-east-1
2025-07-17T11:12:01.8543508Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T11:12:01.8543839Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T11:12:01.8549109Z   AWS_SESSION_TOKEN: ***
2025-07-17T11:12:01.8549489Z   CONTAINER_NAME: 16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a
2025-07-17T11:12:01.8549906Z ##[endgroup]
2025-07-17T11:12:01.8711257Z cat: 'test/**/*_toprint.log': No such file or directory
2025-07-17T11:12:01.8826972Z Prepare all required actions
2025-07-17T11:12:01.8827660Z Getting action download info
2025-07-17T11:12:02.1373950Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2025-07-17T11:12:02.6717359Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-07-17T11:12:03.3267331Z ##[group]Run ./.github/actions/upload-test-artifacts
2025-07-17T11:12:03.3267671Z with:
2025-07-17T11:12:03.3267880Z   use-gha: true
2025-07-17T11:12:03.3268189Z   file-suffix: test-inductor-1-2-linux.rocm.gpu.2_46159470976
2025-07-17T11:12:03.3268555Z   s3-bucket: gha-artifacts
2025-07-17T11:12:03.3268812Z env:
2025-07-17T11:12:03.3269012Z   GIT_DEFAULT_BRANCH: main
2025-07-17T11:12:03.3269399Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T11:12:03.3269970Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T11:12:03.3270567Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T11:12:03.3271466Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T11:12:03.3272234Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T11:12:03.3272497Z   AWS_REGION: us-east-1
2025-07-17T11:12:03.3272812Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T11:12:03.3273154Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T11:12:03.3278512Z   AWS_SESSION_TOKEN: ***
2025-07-17T11:12:03.3278898Z   CONTAINER_NAME: 16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a
2025-07-17T11:12:03.3279320Z ##[endgroup]
2025-07-17T11:12:03.3348237Z ##[group]Run actions/upload-artifact@v4
2025-07-17T11:12:03.3348509Z with:
2025-07-17T11:12:03.3348863Z   name: test-jsons-runattempt1-test-inductor-1-2-linux.rocm.gpu.2_46159470976.zip
2025-07-17T11:12:03.3349289Z   retention-days: 14
2025-07-17T11:12:03.3349532Z   if-no-files-found: warn
2025-07-17T11:12:03.3349954Z   path: test/**/*.json
2025-07-17T11:12:03.3350171Z   compression-level: 6
2025-07-17T11:12:03.3350378Z   overwrite: false
2025-07-17T11:12:03.3350595Z   include-hidden-files: false
2025-07-17T11:12:03.3350824Z env:
2025-07-17T11:12:03.3351004Z   GIT_DEFAULT_BRANCH: main
2025-07-17T11:12:03.3351364Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T11:12:03.3351891Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T11:12:03.3352386Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T11:12:03.3353212Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T11:12:03.3353951Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T11:12:03.3354203Z   AWS_REGION: us-east-1
2025-07-17T11:12:03.3354502Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T11:12:03.3354970Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T11:12:03.3360256Z   AWS_SESSION_TOKEN: ***
2025-07-17T11:12:03.3360642Z   CONTAINER_NAME: 16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a
2025-07-17T11:12:03.3361053Z ##[endgroup]
2025-07-17T11:12:04.0832175Z With the provided path, there will be 29 files uploaded
2025-07-17T11:12:04.0855367Z Artifact name is valid!
2025-07-17T11:12:04.0855742Z Root directory input is valid!
2025-07-17T11:12:04.2922850Z Beginning upload of artifact content to blob storage
2025-07-17T11:12:04.5752920Z Uploaded bytes 48730
2025-07-17T11:12:04.6516757Z Finished uploading artifact content to blob storage!
2025-07-17T11:12:04.6519656Z SHA256 digest of uploaded artifact zip is da2a75b90326374ef99edd66e3d03ca8723fee73a90bf9031b676230bddf2ed2
2025-07-17T11:12:04.6521004Z Finalizing artifact upload
2025-07-17T11:12:04.7886218Z Artifact test-jsons-runattempt1-test-inductor-1-2-linux.rocm.gpu.2_46159470976.zip.zip successfully finalized. Artifact ID 3553780700
2025-07-17T11:12:04.7888112Z Artifact test-jsons-runattempt1-test-inductor-1-2-linux.rocm.gpu.2_46159470976.zip has been successfully uploaded! Final size is 48730 bytes. Artifact ID is 3553780700
2025-07-17T11:12:04.7889929Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16337959866/artifacts/3553780700
2025-07-17T11:12:04.8150440Z ##[group]Run actions/upload-artifact@v4
2025-07-17T11:12:04.8150757Z with:
2025-07-17T11:12:04.8151163Z   name: test-reports-runattempt1-test-inductor-1-2-linux.rocm.gpu.2_46159470976.zip
2025-07-17T11:12:04.8151645Z   retention-days: 14
2025-07-17T11:12:04.8151934Z   if-no-files-found: ignore
2025-07-17T11:12:04.8152214Z   path: test/**/*.xml
test/**/*.csv

2025-07-17T11:12:04.8152507Z   compression-level: 6
2025-07-17T11:12:04.8152757Z   overwrite: false
2025-07-17T11:12:04.8153006Z   include-hidden-files: false
2025-07-17T11:12:04.8153270Z env:
2025-07-17T11:12:04.8153512Z   GIT_DEFAULT_BRANCH: main
2025-07-17T11:12:04.8153912Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T11:12:04.8154489Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T11:12:04.8155012Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T11:12:04.8155882Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T11:12:04.8156665Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T11:12:04.8156935Z   AWS_REGION: us-east-1
2025-07-17T11:12:04.8157257Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T11:12:04.8157604Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T11:12:04.8162923Z   AWS_SESSION_TOKEN: ***
2025-07-17T11:12:04.8163346Z   CONTAINER_NAME: 16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a
2025-07-17T11:12:04.8163783Z ##[endgroup]
2025-07-17T11:12:05.5765961Z With the provided path, there will be 116 files uploaded
2025-07-17T11:12:05.5771211Z Artifact name is valid!
2025-07-17T11:12:05.5772409Z Root directory input is valid!
2025-07-17T11:12:05.7579247Z Beginning upload of artifact content to blob storage
2025-07-17T11:12:06.3083747Z Uploaded bytes 596435
2025-07-17T11:12:06.3576550Z Finished uploading artifact content to blob storage!
2025-07-17T11:12:06.3579498Z SHA256 digest of uploaded artifact zip is 220d5245a31df40e0e2b1503d627974427be38493ee32739b3b24fbc0ca08362
2025-07-17T11:12:06.3580904Z Finalizing artifact upload
2025-07-17T11:12:06.4706664Z Artifact test-reports-runattempt1-test-inductor-1-2-linux.rocm.gpu.2_46159470976.zip.zip successfully finalized. Artifact ID 3553780893
2025-07-17T11:12:06.4708533Z Artifact test-reports-runattempt1-test-inductor-1-2-linux.rocm.gpu.2_46159470976.zip has been successfully uploaded! Final size is 596435 bytes. Artifact ID is 3553780893
2025-07-17T11:12:06.4712905Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16337959866/artifacts/3553780893
2025-07-17T11:12:06.4972036Z ##[group]Run actions/upload-artifact@v4
2025-07-17T11:12:06.4972329Z with:
2025-07-17T11:12:06.4972675Z   name: logs-runattempt1-test-inductor-1-2-linux.rocm.gpu.2_46159470976.zip
2025-07-17T11:12:06.4973087Z   retention-days: 14
2025-07-17T11:12:06.4973336Z   if-no-files-found: ignore
2025-07-17T11:12:06.4973611Z   path: usage_log.txt
test/**/*.log

2025-07-17T11:12:06.4973885Z   compression-level: 6
2025-07-17T11:12:06.4974115Z   overwrite: false
2025-07-17T11:12:06.4974345Z   include-hidden-files: false
2025-07-17T11:12:06.4974590Z env:
2025-07-17T11:12:06.4974807Z   GIT_DEFAULT_BRANCH: main
2025-07-17T11:12:06.4975190Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T11:12:06.4975751Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T11:12:06.4976355Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T11:12:06.4977460Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T11:12:06.4978262Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T11:12:06.4978532Z   AWS_REGION: us-east-1
2025-07-17T11:12:06.4978842Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T11:12:06.4979197Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T11:12:06.4984477Z   AWS_SESSION_TOKEN: ***
2025-07-17T11:12:06.4984878Z   CONTAINER_NAME: 16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a
2025-07-17T11:12:06.4985319Z ##[endgroup]
2025-07-17T11:12:07.2796801Z Multiple search paths detected. Calculating the least common ancestor of all paths
2025-07-17T11:12:07.2798808Z The least common ancestor is /var/home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact
2025-07-17T11:12:07.2799521Z With the provided path, there will be 111 files uploaded
2025-07-17T11:12:07.2804339Z Artifact name is valid!
2025-07-17T11:12:07.2805543Z Root directory input is valid!
2025-07-17T11:12:07.4508523Z Beginning upload of artifact content to blob storage
2025-07-17T11:12:08.2708904Z Uploaded bytes 1632955
2025-07-17T11:12:08.3165266Z Finished uploading artifact content to blob storage!
2025-07-17T11:12:08.3168335Z SHA256 digest of uploaded artifact zip is 7b008b07894b2e6da689d30dd49b49a589a24648075969859ef9613b5c439fe8
2025-07-17T11:12:08.3169820Z Finalizing artifact upload
2025-07-17T11:12:08.4241267Z Artifact logs-runattempt1-test-inductor-1-2-linux.rocm.gpu.2_46159470976.zip.zip successfully finalized. Artifact ID 3553781088
2025-07-17T11:12:08.4243068Z Artifact logs-runattempt1-test-inductor-1-2-linux.rocm.gpu.2_46159470976.zip has been successfully uploaded! Final size is 1632955 bytes. Artifact ID is 3553781088
2025-07-17T11:12:08.4249498Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16337959866/artifacts/3553781088
2025-07-17T11:12:08.4522974Z ##[group]Run # shellcheck disable=SC2156
2025-07-17T11:12:08.4523539Z # shellcheck disable=SC2156
2025-07-17T11:12:08.4524073Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2025-07-17T11:12:08.4551848Z shell: /usr/bin/bash -e {0}
2025-07-17T11:12:08.4552135Z env:
2025-07-17T11:12:08.4552363Z   GIT_DEFAULT_BRANCH: main
2025-07-17T11:12:08.4552783Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T11:12:08.4553387Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T11:12:08.4553944Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T11:12:08.4554829Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T11:12:08.4555649Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T11:12:08.4555940Z   AWS_REGION: us-east-1
2025-07-17T11:12:08.4556406Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T11:12:08.4556769Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T11:12:08.4562246Z   AWS_SESSION_TOKEN: ***
2025-07-17T11:12:08.4562648Z   CONTAINER_NAME: 16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a
2025-07-17T11:12:08.4563102Z ##[endgroup]
2025-07-17T11:12:08.8952581Z ##[group]Run actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
2025-07-17T11:12:08.8953049Z with:
2025-07-17T11:12:08.8953331Z   name: coredumps-inductor-1-2-linux.rocm.gpu.2
2025-07-17T11:12:08.8953691Z   retention-days: 14
2025-07-17T11:12:08.8953952Z   if-no-files-found: ignore
2025-07-17T11:12:08.8954223Z   path: ./**/core.[1-9]*
2025-07-17T11:12:08.8954489Z   compression-level: 6
2025-07-17T11:12:08.8954752Z   overwrite: false
2025-07-17T11:12:08.8954993Z   include-hidden-files: false
2025-07-17T11:12:08.8955252Z env:
2025-07-17T11:12:08.8955474Z   GIT_DEFAULT_BRANCH: main
2025-07-17T11:12:08.8955861Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-17T11:12:08.8956480Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-17T11:12:08.8957009Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-17T11:12:08.8957924Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-17T11:12:08.8958707Z   AWS_DEFAULT_REGION: us-east-1
2025-07-17T11:12:08.8958993Z   AWS_REGION: us-east-1
2025-07-17T11:12:08.8959287Z   AWS_ACCESS_KEY_ID: ***
2025-07-17T11:12:08.8959652Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-17T11:12:08.8964997Z   AWS_SESSION_TOKEN: ***
2025-07-17T11:12:08.8965419Z   CONTAINER_NAME: 16ab158a3199bdb99d9c276861fa88f2fe09ca37781ff9a90e091fbef808622a
2025-07-17T11:12:08.8965881Z ##[endgroup]
2025-07-17T11:12:15.7890012Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded.
2025-07-17T11:12:15.8214323Z Post job cleanup.
2025-07-17T11:12:15.8616756Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-17T11:12:15.8994482Z Post job cleanup.
2025-07-17T11:12:16.0379071Z Post job cleanup.
2025-07-17T11:12:16.0422051Z Post job cleanup.
2025-07-17T11:12:16.1461972Z [command]/usr/bin/git version
2025-07-17T11:12:16.1506137Z git version 2.34.1
2025-07-17T11:12:16.1551647Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/970889d7-882c-4f70-b469-7c69d9050863/.gitconfig'
2025-07-17T11:12:16.1561750Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/970889d7-882c-4f70-b469-7c69d9050863' before making global git config changes
2025-07-17T11:12:16.1563179Z Adding repository directory to the temporary git global config as a safe directory
2025-07-17T11:12:16.1566093Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-07-17T11:12:16.1615835Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-07-17T11:12:16.1653912Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-07-17T11:12:16.2049815Z Entering 'android/libs/fbjni'
2025-07-17T11:12:16.2125889Z Entering 'third_party/FP16'
2025-07-17T11:12:16.2200767Z Entering 'third_party/FXdiv'
2025-07-17T11:12:16.2275014Z Entering 'third_party/NNPACK'
2025-07-17T11:12:16.2343207Z Entering 'third_party/NVTX'
2025-07-17T11:12:16.2413909Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-17T11:12:16.2482361Z Entering 'third_party/XNNPACK'
2025-07-17T11:12:16.2574200Z Entering 'third_party/aiter'
2025-07-17T11:12:16.2647395Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-17T11:12:16.2729388Z Entering 'third_party/benchmark'
2025-07-17T11:12:16.2801301Z Entering 'third_party/composable_kernel'
2025-07-17T11:12:16.2883241Z Entering 'third_party/cpp-httplib'
2025-07-17T11:12:16.2955430Z Entering 'third_party/cpuinfo'
2025-07-17T11:12:16.3032352Z Entering 'third_party/cudnn_frontend'
2025-07-17T11:12:16.3112205Z Entering 'third_party/cutlass'
2025-07-17T11:12:16.3204996Z Entering 'third_party/fbgemm'
2025-07-17T11:12:16.3282158Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-17T11:12:16.3351278Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-17T11:12:16.3418426Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-17T11:12:16.3483279Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-17T11:12:16.3563385Z Entering 'third_party/fbgemm/external/googletest'
2025-07-17T11:12:16.3634431Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-17T11:12:16.3710142Z Entering 'third_party/fbgemm/external/json'
2025-07-17T11:12:16.3779979Z Entering 'third_party/flash-attention'
2025-07-17T11:12:16.3849717Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-17T11:12:16.3924527Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-17T11:12:16.4009854Z Entering 'third_party/flatbuffers'
2025-07-17T11:12:16.4082564Z Entering 'third_party/fmt'
2025-07-17T11:12:16.4157718Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-17T11:12:16.4233660Z Entering 'third_party/gloo'
2025-07-17T11:12:16.4303518Z Entering 'third_party/googletest'
2025-07-17T11:12:16.4374559Z Entering 'third_party/ideep'
2025-07-17T11:12:16.4444544Z Entering 'third_party/ideep/mkl-dnn'
2025-07-17T11:12:16.4525773Z Entering 'third_party/ittapi'
2025-07-17T11:12:16.4596488Z Entering 'third_party/kineto'
2025-07-17T11:12:16.4667066Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-17T11:12:16.4732264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-17T11:12:16.4798384Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-17T11:12:16.4865373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-17T11:12:16.4932347Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-17T11:12:16.4995243Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-17T11:12:16.5060744Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-17T11:12:16.5124826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-17T11:12:16.5189060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-17T11:12:16.5266122Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-17T11:12:16.5331363Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-17T11:12:16.5397480Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-17T11:12:16.5470682Z Entering 'third_party/kleidiai'
2025-07-17T11:12:16.5547237Z Entering 'third_party/mimalloc'
2025-07-17T11:12:16.5626508Z Entering 'third_party/nlohmann'
2025-07-17T11:12:16.5708206Z Entering 'third_party/onnx'
2025-07-17T11:12:16.5804536Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-17T11:12:16.5884603Z Entering 'third_party/opentelemetry-cpp'
2025-07-17T11:12:16.5956317Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-17T11:12:16.6026286Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-17T11:12:16.6087640Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-17T11:12:16.6149989Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-17T11:12:16.6227039Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-17T11:12:16.6283363Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-17T11:12:16.6349958Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-17T11:12:16.6413486Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-17T11:12:16.6480983Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-17T11:12:16.6553495Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-17T11:12:16.6644435Z Entering 'third_party/pocketfft'
2025-07-17T11:12:16.6717417Z Entering 'third_party/protobuf'
2025-07-17T11:12:16.6791736Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-17T11:12:16.6867422Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-17T11:12:16.6933423Z Entering 'third_party/psimd'
2025-07-17T11:12:16.7001382Z Entering 'third_party/pthreadpool'
2025-07-17T11:12:16.7068735Z Entering 'third_party/pybind11'
2025-07-17T11:12:16.7137858Z Entering 'third_party/python-peachpy'
2025-07-17T11:12:16.7207153Z Entering 'third_party/sleef'
2025-07-17T11:12:16.7279365Z Entering 'third_party/tensorpipe'
2025-07-17T11:12:16.7346855Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-17T11:12:16.7410071Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-17T11:12:16.7478499Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-17T11:12:16.7549212Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-17T11:12:16.7604846Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-17T11:12:16.7706695Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-07-17T11:12:16.7738690Z http.https://github.com/.extraheader
2025-07-17T11:12:16.7749562Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2025-07-17T11:12:16.7792876Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-07-17T11:12:16.8176000Z Entering 'android/libs/fbjni'
2025-07-17T11:12:16.8216663Z http.https://github.com/.extraheader
2025-07-17T11:12:16.8273662Z Entering 'third_party/FP16'
2025-07-17T11:12:16.8316514Z http.https://github.com/.extraheader
2025-07-17T11:12:16.8369570Z Entering 'third_party/FXdiv'
2025-07-17T11:12:16.8404736Z http.https://github.com/.extraheader
2025-07-17T11:12:16.8459618Z Entering 'third_party/NNPACK'
2025-07-17T11:12:16.8499621Z http.https://github.com/.extraheader
2025-07-17T11:12:16.8554889Z Entering 'third_party/NVTX'
2025-07-17T11:12:16.8595276Z http.https://github.com/.extraheader
2025-07-17T11:12:16.8651150Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-17T11:12:16.8693840Z http.https://github.com/.extraheader
2025-07-17T11:12:16.8751885Z Entering 'third_party/XNNPACK'
2025-07-17T11:12:16.8784651Z http.https://github.com/.extraheader
2025-07-17T11:12:16.8856515Z Entering 'third_party/aiter'
2025-07-17T11:12:16.8897093Z http.https://github.com/.extraheader
2025-07-17T11:12:16.8954079Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-17T11:12:16.8996345Z http.https://github.com/.extraheader
2025-07-17T11:12:16.9063965Z Entering 'third_party/benchmark'
2025-07-17T11:12:16.9098595Z http.https://github.com/.extraheader
2025-07-17T11:12:16.9155108Z Entering 'third_party/composable_kernel'
2025-07-17T11:12:16.9197799Z http.https://github.com/.extraheader
2025-07-17T11:12:16.9261977Z Entering 'third_party/cpp-httplib'
2025-07-17T11:12:16.9303972Z http.https://github.com/.extraheader
2025-07-17T11:12:16.9356640Z Entering 'third_party/cpuinfo'
2025-07-17T11:12:16.9398901Z http.https://github.com/.extraheader
2025-07-17T11:12:16.9455515Z Entering 'third_party/cudnn_frontend'
2025-07-17T11:12:16.9488346Z http.https://github.com/.extraheader
2025-07-17T11:12:16.9544124Z Entering 'third_party/cutlass'
2025-07-17T11:12:16.9586797Z http.https://github.com/.extraheader
2025-07-17T11:12:16.9647245Z Entering 'third_party/fbgemm'
2025-07-17T11:12:16.9687271Z http.https://github.com/.extraheader
2025-07-17T11:12:16.9745577Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-17T11:12:16.9781110Z http.https://github.com/.extraheader
2025-07-17T11:12:16.9827466Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-17T11:12:16.9862152Z http.https://github.com/.extraheader
2025-07-17T11:12:16.9921591Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-17T11:12:16.9957301Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0005921Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-17T11:12:17.0041243Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0102085Z Entering 'third_party/fbgemm/external/googletest'
2025-07-17T11:12:17.0137387Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0186359Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-17T11:12:17.0218541Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0264146Z Entering 'third_party/fbgemm/external/json'
2025-07-17T11:12:17.0300814Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0355500Z Entering 'third_party/flash-attention'
2025-07-17T11:12:17.0397375Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0452298Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-17T11:12:17.0489104Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0553534Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-17T11:12:17.0589542Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0658009Z Entering 'third_party/flatbuffers'
2025-07-17T11:12:17.0699913Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0754541Z Entering 'third_party/fmt'
2025-07-17T11:12:17.0793815Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0845404Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-17T11:12:17.0880648Z http.https://github.com/.extraheader
2025-07-17T11:12:17.0932368Z Entering 'third_party/gloo'
2025-07-17T11:12:17.0968331Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1026165Z Entering 'third_party/googletest'
2025-07-17T11:12:17.1062947Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1117525Z Entering 'third_party/ideep'
2025-07-17T11:12:17.1157157Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1207635Z Entering 'third_party/ideep/mkl-dnn'
2025-07-17T11:12:17.1241933Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1301656Z Entering 'third_party/ittapi'
2025-07-17T11:12:17.1342016Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1390832Z Entering 'third_party/kineto'
2025-07-17T11:12:17.1431744Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1484517Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-17T11:12:17.1520321Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1573250Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-17T11:12:17.1611706Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1669847Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-17T11:12:17.1712288Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1765257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-17T11:12:17.1799699Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1855110Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-17T11:12:17.1892716Z http.https://github.com/.extraheader
2025-07-17T11:12:17.1945950Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-17T11:12:17.1981106Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2032222Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-17T11:12:17.2073502Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2125129Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-17T11:12:17.2161436Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2217496Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-17T11:12:17.2253280Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2302272Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-17T11:12:17.2336698Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2394363Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-17T11:12:17.2432138Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2484822Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-17T11:12:17.2521965Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2575446Z Entering 'third_party/kleidiai'
2025-07-17T11:12:17.2613273Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2661811Z Entering 'third_party/mimalloc'
2025-07-17T11:12:17.2702997Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2759316Z Entering 'third_party/nlohmann'
2025-07-17T11:12:17.2801269Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2855066Z Entering 'third_party/onnx'
2025-07-17T11:12:17.2913415Z http.https://github.com/.extraheader
2025-07-17T11:12:17.2971661Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-17T11:12:17.3011919Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3066390Z Entering 'third_party/opentelemetry-cpp'
2025-07-17T11:12:17.3099130Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3150845Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-17T11:12:17.3186878Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3240171Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-17T11:12:17.3277563Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3328136Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-17T11:12:17.3364024Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3412636Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-17T11:12:17.3443327Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3494141Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-17T11:12:17.3525334Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3578683Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-17T11:12:17.3612707Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3662040Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-17T11:12:17.3697348Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3751141Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-17T11:12:17.3787812Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3838012Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-17T11:12:17.3877840Z http.https://github.com/.extraheader
2025-07-17T11:12:17.3935507Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-17T11:12:17.3974890Z http.https://github.com/.extraheader
2025-07-17T11:12:17.4057255Z Entering 'third_party/pocketfft'
2025-07-17T11:12:17.4097344Z http.https://github.com/.extraheader
2025-07-17T11:12:17.4153841Z Entering 'third_party/protobuf'
2025-07-17T11:12:17.4198334Z http.https://github.com/.extraheader
2025-07-17T11:12:17.4252795Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-17T11:12:17.4286576Z http.https://github.com/.extraheader
2025-07-17T11:12:17.4341472Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-17T11:12:17.4378040Z http.https://github.com/.extraheader
2025-07-17T11:12:17.4433278Z Entering 'third_party/psimd'
2025-07-17T11:12:17.4477785Z http.https://github.com/.extraheader
2025-07-17T11:12:17.4530424Z Entering 'third_party/pthreadpool'
2025-07-17T11:12:17.4567705Z http.https://github.com/.extraheader
2025-07-17T11:12:17.4618340Z Entering 'third_party/pybind11'
2025-07-17T11:12:17.4658320Z http.https://github.com/.extraheader
2025-07-17T11:12:17.4711867Z Entering 'third_party/python-peachpy'
2025-07-17T11:12:17.4749925Z http.https://github.com/.extraheader
2025-07-17T11:12:17.4803247Z Entering 'third_party/sleef'
2025-07-17T11:12:17.4852734Z http.https://github.com/.extraheader
2025-07-17T11:12:17.4915536Z Entering 'third_party/tensorpipe'
2025-07-17T11:12:17.4959118Z http.https://github.com/.extraheader
2025-07-17T11:12:17.5012532Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-17T11:12:17.5050559Z http.https://github.com/.extraheader
2025-07-17T11:12:17.5105017Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-17T11:12:17.5139617Z http.https://github.com/.extraheader
2025-07-17T11:12:17.5187738Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-17T11:12:17.5229009Z http.https://github.com/.extraheader
2025-07-17T11:12:17.5281046Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-17T11:12:17.5317167Z http.https://github.com/.extraheader
2025-07-17T11:12:17.5365099Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-17T11:12:17.5396741Z http.https://github.com/.extraheader
2025-07-17T11:12:17.5652065Z Cleaning up orphan processes